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Abstract 

The  use  of  some  degree  of  concurrency  in  weapon  system 
acquisition  has  become  a  normal  mode  of  operation.  There 
are  several  benefits  and  problems  associated  with  concurrent 
programs.  The  recent  elevation  of  reliability  and  maintain¬ 
ability  (R&M)  to  a  status  equal  to  performance,  cost,  and 
schedule  when  evaluating  current  weapon  systems  has  added  to 
the  list  of  potential  problems  experienced  by  concurrent 
programs. 

A  literature  review  was  conducted  which  traced  the 
history  of  concurrency  from  the  Ballistic  Missile  Programs 
to  the  1986  Packard  Commission  Report.  This  review  focused 
on  the  reasons  for  the  continued  discussions  on  the  overall 
value  of  concurrency.  The  review  also  looked  at  the  impact 
of  concurrency  on  system  R&M.  Several  factors  were  identi¬ 
fied  which  existed  in  concurrent  programs  and  showed  a 
potential  to  limit  system  R&M.  In  addition  the  study 
covered  the  causes  for  the  variances  between  the  system  R&M 
measures  demonstrated  in  the  developmental ,and  operational 
environments. 

The  researcher  interviewed  fifteen  managers  who  were 
involved  in  five  concurrent  programs.  These  managers  were 
from  the  following  areas:  the  System  Program  Office,  Deputy 
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Program  Manager  for  Logistics,  and  the  Air  Force  Office  for 
Test  and  Evaluation.  The  interviews  focused  on  the  opinions 
of  the  managers  on  concurrency's  use  and  how  it  affected  R&M 
development  in  their  program. 

The  results  of  this  study  indicate  that  concurrency 
does  impact  system  R&M  development.  However,  the  amount  of 
impact  and  the  applicability  of  the  factors  reviewed  varies 
by  program.  Managers  '  opinions  of  the  factors  appear  to  be 
influenced  by  their  position  in  the  acquisition  program. 

The  benefits  and  problems  of  concurrency  are  covered.  The 
causes  for  the  disparity  between  field  and  development  R&M 
measures,  suggestions  to  correct  this  R&M  problem,  and 
recommendations  to  improve  system  R&M  are  discussed. 
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THE  USE  OF  CONCURRENCY 


IN  THE  ACQUISITION  PROCESS 
AND  ITS  IMPACT  ON 
RELIABILITY  AND  MAINTAINABILITY 


I  •  Introduct ion 


General  Issue 

In  June  1085,  the  General  Accounting  Office  CGAO) 
released  a  report  titled  "Production  of  Some  Major  Weapon 
Systems  Began  With  Only  Limited  Operational  Test  and  Evalua¬ 
tion  Results."  The  report  identified  nine  systems  and 
claimed  that  the  high  use  of  test  concurrency  had  produced 
inconclusive  data  to  prove  system  adequacy  (5:44).  While 
the  Department  of  Defense  (DOD)  partially  concurred  with  the 
GAO  findings,  DOD  maintained  that  the  United  States'  acqui¬ 
sition  process  usually  produces  highly  capable  weapon 
systems  (2:41).  Test  concurrency  is  being  used  more  anc. 
more  in  weapon  system  development  (5:43).  Does  concurrency 
result  in  inadequate  operational  test  and  evaluation 
results? 

Background 

The  use  of  concurrency  in  the  DOD  acquisition  process 
has  sparked  controversy  for  the  last  three  decades.  While  a 
form  of  concurrency  had  been  used  by  industry,  the  first  use 
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of  the  term  by  DOD  seems  to  have  occurred  in  1958  with  the 

Ballistic  Missile  Program  (23:67).  A  1959  report  titled 

"The  United  States  Guided  Missile  Program"  prepared  for  the 

Senate  Armed  Services  Committee  addressed  concurrency  as: 

(The  Air  Force! . . .  has  adopted  and  expanded  another 
technique  often  used  by  industry  where  competition 
in  getting  to  a  market  is  keen;  that  is  compressing 
the  periods  of  development  of  new  products  and 
getting  production  started.  In  the  case  of  the 
missile  program  .  .  .  the  Air  Force  is  undertaking  to 
do  this  by  what  they  call  the  "concept  of 
concurrency"  (23:67). 

The  concept  of  concurrency  was  teamwork  applied  with  modern 

management  techniques.  The  use  of  concurrency  would  provide 

...an  overlapping  of  the  development  functions  so 
that,  for  instance,  flight  test  can  proceed 
coincident  with  production,  construction  can  get 
underway  while  flight  test  is  in  progress,  and 
training  can  be  initiated  concurrently  with  testing 
and  production  (42:238). 

Besides  the  Ballistic  Missile  Program,  the  U-2  and  SR-71 
aircraft  were  developed  using  concurrency.  The  successful 
development  of  these  programs  brought  the  use  of  concurrency 
solidly  into  the  DOD  acquisition  process  (16:42;  28:56). 

The  success  of  concurrency  was  short  lived.  Many 
programs  developed  during  the  1960s  experienced  both  cost 
and  performance  problems  (7:48;  16:52).  Deputy  Secretary  of 
Defense  David  Packard  identified  several  acquisition  prob¬ 
lems  in  systems  developed  prior  to  1969.  These  were  cost 
overruns,  excessive  time  from  conception  to  delivery,  and 
low  reliability  (40:3).  In  almost  every  program  where  pro¬ 
duction  was  started  before  development  and  testing  was 
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completed,  both  money  and  time  was  wasted  (12:4).  Since 
production  was  started  before  completion  of  development, 
costly  engineering  changes  had  to  be  accomplished  on  the 
production  line  (40:4).  In  1978  the  Defense  Science  Board 
defined  concurrency  as: 

The  conduct  of  steps  leading  to  production  before 
the  end  of  full  scale  development  time  span.  The 
steps  referred  to  include:  manufacturing  planning, 
process  development,  tool  and  test  equipment  design, 
and  fabrication  and  ordering  of  long- lead  materials 
(7:47). 

Concurrency  was  identified  as  the  cause  for  these  cost  and 

performance  problems  (23:68;  40:3). 

Consequently,  Mr.  Packard  introduced  an  acquisition 

policy  which  rejected  the  use  of  concurrency  and  required 

the  use  of  prototypes.  The  DOD  would  use  a  sequential 

acquisition  process  (16:53-54).  Weapon  system  acquisition 

would  be  accomplished  in  three  phases:  program  initiation, 

full  scale  development,  and  production/development 

(35=22-24).  According  to  an  article  by  Robert  Gibson,  a 

1971  Rand  report  recommended: 

A  sequential  approach  to  major  system  acquisitions 
with  clearly  defined  milestones.  The  normal 
strategy  for  system  acquisition  in  the  1970s  should 
involve  a  conscious  decision  to  produce  (or  not 
produce)  only  after  the  development  is  complete 
(23:68). 

David  Packard's  acquisition  policies  failed  to  correct 
all  the  problems  of  the  DOD  acquisition  system.  Due  to  an 
increasing  acquisition  time  cycle,  a  Defense  Science  Board 
report  in  1978  stated: 
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That  the  acquisition  process  has  gone  to  unreason¬ 
able  limits  in  discouraging  concurrency  and  in  over 
emphasizing  advanced  development  prototypes  even 
when  these  add  more  to  program  cost  and  acquisition 
time  than  they  benefit  it  by  reducing  risk  (7=V). 

Although  considered  successful,  prototype  programs  such  as 
the  F-16  and  the  A-10  resulted  in  acquisition  times  of  nine 
and  eleven  years,  respectively  (52=110).  Concern  was 
expressed  over  the  growing  length  of  time  to  develop  new 
systems.  The  acquisition  cycle  was  taking  from  12  to  15 
years  to  produce  new  systems  while  modern  commercial  aircraft 
were  being  developed  in  about  8  years  C 47: 3-4).  Many  of 
today’s  weapon  system  acquisition  programs  use  a  combination 
of  concurrency  and  prototyping  (54:40-41).  A  review  of 
Government  Accounting  Office  reports  showed  that  DOD  acqui¬ 
sition  programs  continue  to  experience  problems  (19;  20;  21). 
Current  acquisition  problems  have  been  identified  as  exces¬ 
sive  time  and  cost,  along  with  the  need  to  improve  perform¬ 
ance  and  readiness  (1:2;  16=68-70). 

Specif ic  Problem 

Concurrency  has  become  a  factor  in  almost  all  major 
weapon  system  acquisitions.  Reviews  of  the  acquisition 
process  have  been  accomplished  to  determine  the  reasons  for 
continual  problems.  Numerous  policy  and  organizational 
changes  have  been  implemented  to  make  the  DOD  process  more 
efficient,  cost  effective,  and  to  shorten  the  length  of  the 
acquisition  cycle.  Actions  to  reduce  acquisition  times  may 
conflict  with  steps  to  improve  system  reliability  (47=9). 
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Since  current  system  development  programs  take  up  to  ten 
years  or  more,  the  increased  emphasis  on  reliability  and 
maintainability  CR&M)  initiated  in  the  1980s  has  not  been 
conclusively  evaluated-  There  is  a  requirement  to  determine 
if  the  use  of  concurrency  during  system  acquisition  and  the 
need  for  improved  R&M  are  compatible.  What  impact  does 
concurrency  have  on  our  acquisition  programs  and  system  R&M? 

Scope  of  Research 

This  study  will  review  the  use  of  concurrency  in  system 
acquisition.  A  literature  review  will  be  accomplished  to 
identify  the  reasons  for  the  fluctuating  management  support 
of  concurrency.  Interviews  will  be  conducted  with  managers 
involved  in  the  acquisition  process  from  Air  Force  Systems 
Command  CAFSC),  Air  Force  Logistics  Command  CAFLC),  and  the 
Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC). 
These  interviews  will  provide  information  from  three 
different  management  perspectives.  The  managers  from  AFSC 
will  provide  data  from  an  overall  system  responsibility 
aspect,  AFLC  provides  a  primary  focus  on  logistics  support, 
and  AFOTEC  will  provide  information  from  the  operational 
testing  standpoint.  This  study  will  focus  on  reliability 
and  maintainability  during  system  development. 

Research  Questions 

During  the  analysis  of  concurrent  programs,  the 
researcher  will  answer  the  following  questions: 
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1.  What  weapon  system  programs  were  successfully  developed 
using  concurrency? 

2.  Why  has  the  use  of  concurrency  been  periodically 
accepted  and  rej  ected? 

3.  What  are  the  resulting  benefits  of  using  concurrency? 

4.  What  are  the  potential  problems  associated  with  the  use 
of  concurrency? 

5.  How  well  do  the  quantitative  R&M  indicators  developed 
during  testing  predict  field  R&M  experience? 

The  interviews  of  acquisition  managers  will  be  designed 
to  determine: 

6.  How  does  the  use  of  concurrency  affect  acquisition 
programs? 

7.  How  does  concurrency  impact  system  reliability  and 
maintainabi 1 ity? 

8.  What  can  be  done  to  improve  system  R&M  in  the  acquisi¬ 
tion  process? 

Summary 

The  attempts  to  correct  acquisition  problems  have 
centered  around  two  main  factions.  Air  Force  and  DOD 
managers  have  been  divided  into  two  groups,  those  who  favor 
accelerated  development  programs  and  those  who  want  a 
sequential  process.  This  difference  of  opinion  has  resulted 
in  policies  and  directives  from  one  extreme  to  another 
either  supporting  concurrency  or  rejecting  it  C8=I-2). 

Col  Bradson  in  an  article  for  Program  Manager  explained 
the  need  for  concurrency  was  due  to  the  rapid  development  of 
new  technologies.  Current  acquisition  programs  taking  8  to 
15  years  to  complete  will  span  from  2  to  4  technological 
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generations  (3:12).  There  are  some  people  in  the  acquisi¬ 
tion  world  who  believe  we  have  always  used  some  degree  of 
concurrency.  Deputy  Secretary  of  Defense  Paul  Thayer 
reportedly  stated: 

"The  old  myth,  fly  before  buy,  was  Just  that,"... 

"It  is  a  myth.  It 's  never  been  practiced  in  pure 
form.  There  has  always  been  some  concurrency  in 
any  program  I'm  aware  of.  If  you  did  wait  until 
development  and  operational  testing  was  complete 
before  going  on  to  large  scale  production,  the 
system  would  be  obsolete"  (2:26). 

Today's  acquisition  process  uses  concurrency.  The  degree  of 
concurrency  is  dependent  on  the  individual  program's  risk 
and  how  soon  it  is  needed  (37:46).  A  review  of  the  litera¬ 
ture  to  determine  concurrency 's  affect  on  acquisition  and 
R&M  is  warranted. 


II.  Literature  Review 


The  purpose  of  this  chapter  is  to  review  the  use  of 
concurrency  in  the  Department  of  Defense  CDOD)  weapon  system 
acquisition  process  and  to  look  at  the  emergence  of  relia¬ 
bility  and  maintainability  as  performance  factors.  The 
weapon  system  acquisition  process  is  defined  as: 

A  sequence  of  specified  decision  events  and  phases 
of  activity  directed  to  achievement  of  established 
program  objectives  in  the  acquisition  of  Defense 
Systems  and  extending  from  approval  of  a  mission 
need  through  successful  deployment  of  the  Defense 
System  or  termination  of  the  program  C31:l). 


Concurrency 

The  term  concurrency  has  had  many  definitions  since  its 
origin  in  the  1950s.  According  to  Captain  Wayne  Foote,  in  a 
Management  Consulting  and  Research,  Inc.  report  titled 
"Shortening  the  Acquisition  Cycle:  Research  on  Concurrency” 
concurrency  is  interpreted  as: 

1)  parallel  (back-up)  technology  development 

2)  simultaneous,  but  independent,  technological 
development  and  testing, 

3)  co-production,  and 

4)  overlap  of  dependent,  normally  sequential 
activities  C16:3). 

In  the  area  of  the  weapon  system  acquisition  process, 
concurrency  can  be  defined  as  a  strategy  which  results  in 
the  overlap  of  some  or  all  the  process  phases.  Today,  these 
phases  are  concept  exploration,  demonstration  and  valida¬ 
tion,  full  scale  development,  and  production  and  deployment. 
Varying  degrees  of  concurrency  have  been  used  in  DOD  acqui- 


sition  programs  for  the  past  thiry  years.  During  this  time 
frame,  concurrency  has  been  credited  with  both  fixing  and 
causing  many  of  the  problems  in  DOD  system  acquisition.  A 
historical  review  of  some  of  the  programs  and  outcomes 
resulting  from  the  use  of  concurrency  follows. 

Concurrency :  Initial  Successes 

Concurrency  had  its  origin  with  the  advent  of  the 
nuclear  age  and  the  successes  of  the  German  and  Soviet  Union 
scientists  in  rocket  propulsion.  Several  senior  civilian 
and  military  leaders  in  the  United  States  realized  that  the 
advantages  of  time  and  distance  which  enabled  them  to  pre¬ 
pare  for  World  War  II  could  not  be  guaranteed  in  a  future 
war.  Also,  the  potential  of  a  Soviet  Union  technological 
breakthrough  which  would  produce  the  first  intercontinental 
ballistic  missile  (ICBM)  posed  a  serious  threat  to  national 
security  Consequently,  an  urgent  need  surfaced  to  develop 
a  better  and  quicker  weapon  system  acquisition  process 
C  42 : 238) • 

To  meet  the  potential  threat  to  national  security  the 
Ballistic  Missile  Program  was  initiated.  This  program  was 
to  be  managed  using  a  new  acquisition  strategy  called 
"concurrency"  (49:12).  The  concept  of  concurrency  was  team¬ 
work  applied  with  modern  management  techniques.  Many  renior 
leaders  felt  this  new  strategy  would  allow  the  United  States 
to  develop  an  ICBM  before  the  Soviet  Union.  The  use  of 
concurrency  would  provide  for: 
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...an  overlapping  of  the  development  functions  so 
that,  for  instance,  flight  test  can  proceed  coinci¬ 
dent  with  production,  construction  can  get  underway 
while  flight  test  is  in  progress,  and  training  can 
be  initiated  concurrently  with  testing  and  produc¬ 
tion  (42:238). 

The  Ballistic  Missile  Program  was  responsible  for  developing 
three  missile  configurations:  Atlas,  Titan,  and  Thor.  Due 
to  the  numerous  emerging  technologies  associated  with  devel¬ 
opment  of  the  first  ICBM  and  the  normal  acquisition  proce¬ 
dures,  the  missile  development  was  expected  to  take  from  10 
to  11  years  (42:244).  This  projection  was  the  catalyst  in 
the  decision  to  use  concurrency  in  the  Ballistic  Missile 
Program.  A  concurrent  effort  would  shorten  lead  time  in  the 
acquisition  process.  By  reducing  lead  time,  these  new 
missiles  would  have  an  extended  life  expectancy  and  be  less 
susceptible  to  technological  obsolescence.  The  Ballistic 
Missile  Program  was  considered  extremely  successful  with  the 
Atlas  ICBM  being  developed  and  deployed  within  five  years  of 
program  startup.  The  Thor,  intermediate  range  missile,  was 
even  more  impressive  as  it  achieved  operational  status  in 
four  years  (42:240,250). 

Several  management  decisions  contributed  to  the  overall 
success  of  the  ICBM  program.  The  creation  of  an  autonomous 
organization,  the  Ballistic  Missile  Division,  was  a  signif¬ 
icant  deviation  from  the  normal  acquisition  process.  This 
organization  had  overall  responsibility  and  authority  for 
all  aspects  of  the  ICBM  program.  Management  guidelines  were 
established  to  provide  for  maximum  priority  and  decentral i- 
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zation,  minimum  committee  operation  and  red  tape,  and  a 
level  of  authority  commensurate  with  responsibility 
C 16= 17- 18).  This  structure  allowed  management  decisions  to 
be  made  at  the  lowest  capable  level.  The  organization  staff 
consisted  of  a  small  number  of  highly  qualified  personnel. 

To  provide  additional  highly  skilled  scientific  and  techni¬ 
cal  personnel,  an  engineering  group  was  obtained  through  a 
contract  with  the  Guided  Missile  Research  Division  of  the 
Ramo-Woo ldridge  Corporation.  Another  important  element  in 
the  ICBM  program  was  the  use  of  competition  during  develop¬ 
ment  (16:20-23). 

Other  extremely  successful  acquisition  programs  were 
developed  during  the  1950s  using  similar  management  tech¬ 
niques  as  the  Ballistic  Missile  Program.  Two  well  known 
examples  are  the  U-2  and  SR- 71  aircraft.  According  to  Dr. 
Richard  P.  Hal  lion,  Lockheed's  success  was  possible: 

Because  of  more  streamlined  management,  smaller 
development  teams,  stringent  review  of  mission 
requirements,  and  rigorous  adherence  to  cost  and 
time  schedules  (28:56). 

Additional  management  principle  similarities  between  the 
ICBM  program  and  Lockheed's  programs  were: 

1.  Program  manager  had  practically  complete  control. 

2.  Highly  skilled  personnel  in  project  offices. 

3.  High  level  of  inspection  and  testing  by 
subcontractor  and  vendor.  Limit  duplication  of 
inspection  and  testing. 

4.  Timely  funding. 
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5.  Mutual  trust  between  military  and  contractor 
personnel  fostering  a  close  working  relationship. 

6.  Limit  access  to  the  project  and  its  personnel 
through  appropriate  security  measures  C 16: 44-46). 

The  successful  development  of  the  U-2,  SR-71,  and  ICBMs 
brought  the  use  of  concurrency  solidly  into  the  DOD  acquisi¬ 
tion  process  C 16: 43-44;  28:56).  However,  many  subsequent 
concurrent  programs  faced  significant  difficulties. 

Concurrency  Problems 

Some  of  the  first  programs  that  demonstrated  problems 
in  using  concurrency  were  the  early  cruise  missile  programs; 
the  Snark  and  Navaho.  These  programs  suffered  from  schedule 
slippage  and  technical  problems.  In  the  final  outcome  the 
Snark  was  completed  six  years  behind  schedule  and  Navaho  was 
cancelled  after  a  three  year  slip  C 16:28).  The  reasons  for 
the  failure  of  the  cruise  missile  programs  can  be  found  in 
management  philosophy  differences  when  compared  to  the 
Ballistic  Missile  Program.  Concurrency  was  introduced  into 
the  cruise  missile  programs  through  unplanned  schedule 
compression.  Additionally,  the  program  managers  were  not 
given  the  same  autonomy,  responsibility,  and  authority  as  in 
the  ICBM  program  (16:26-29). 

From  1958  to  1970  acquisition  programs  were  being 
developed  with  a  concept  called  "category  testing" 
(54:17-19).  This  testing  consisted  of  two  phases  of  devel¬ 
opment  test  and  evaluation  (DT&E)  and  a  third  phase  covering 
operational  test  and  evaluation  (OT&E)  (1:6).  According  to 
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Major  Adams,  concurrency  was  used  to  expedite  the  overall 
acquisition  process  through  an  overlap  of  the  two  DT&E 
phases.  The  OT&E  phase  was  not  conducted  until  after  the 
production  decision  and  the  availability  of  production 
aircraft  C 1:6-7;  49:9). 

In  the  1960s  concurrency  became  more  entrenched  in  the 
acquisition  cycle  with  the  practice  of  Total  Package 
Procurement  C7.-47).  Total  Package  Procurement  required  a 
governmental  commitment  to  production  of  a  system  at  the 
time  of  contract  award  C54:43).  DOD  continued  to  move  away 
from  the  principles  which  had  allowed  the  Ballistic  Missile 
Program  to  be  successful.  In  an  effort  to  streamline  the 
acquisition  process,  Air  Force  Systems  Command  and  Air  Force 
Logistics  Command  were  created.  However,  this  reorganiza¬ 
tion  was  not  able  to  reduce  the  levels  of  decision  making  or 
regain  control  of  systems  engineering  from  the  contractor 
and  place  it  back  into  the  project  office  C36=17).  Concur¬ 
rency  had  been  very  effective  with  a  maximum  decentraliza¬ 
tion  of  authority  and  responsibility  to  the  program  manager 
level-  The  increasing  centralization  of  authority  and 
layers  of  management  were  making  the  acquisition  process 
ineffective.  Many  systems  under  development  were  experi¬ 
encing  both  cost  and  performance  problems  C48=2).  The 
MBT-70,  Main  Battle  Tank,  and  F-111B  programs  were  cancelled 
after  considerable  expenditures.  While  the  C-5A  and  F-lll 
programs  continued  to  production,  they  had  large  cost  over- 
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runs  and  performance  problems  (7:48;  49:12).  Concurrency 
was  identified  as  the  cause  of  these  acquisition  problems 
(7:48;  16:52).  Also,  concurrency  resulted  in  new  aircraft 
with  known  deficiencies  being  delivered  to  operational 
units.  These  aircraft  were  flown  under  some  restrictions 
until  engineering  fixes  could  be  designed,  tested,  and 
incorporated.  This  is  what  happened  in  the  C-141  program 
because  of  a  problem  with  the  central  air  data  computer 
(54:25-26).  The  1970s  brought  a  move  away  from  the  use  of 
wholesale  concurrency. 


Prototypes 

Deputy  Secretary  of  Defense  David  Packard  introduced  an 

acquisition  policy  which  rejected  the  use  of  concurrency  and 

required  the  use  of  prototypes.  DOD  would  use  a  sequential 

acquisition  process  (16=53-54). 

In  July  1970,  the  blue  ribbon  defense  panel  recom¬ 
mended,  among  other  things,  the  following:  More  use 
of  competitive  prototypes  and  less  reliance  on 
paper  studies.  Selected  lengthening  of  production 
schedules,  keeping  the  system  in  production  over  a 
greater  period  of  time  so  that  incremental  improve¬ 
ments  could  be  introduced.  A  general  rule  against 
concurrent  development  and  production  efforts, with 
the  production  decision  deferred  until  successful 
demonstration  of  developmental  prototypes  (7=48). 

From  1970  to  1977  there  were  continual  discussions  and 

congressional  reviews  on  the  use  of  concurrency.  According 

to  Captain  Foote,  the  military  services  were  not  ready  to 

accept  the  increased  acquisition  time  and  costs  associated 

with  sequential  acquisition.  After  rejecting  the  use  of 
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concurrency,  Mr-  Packard  endorsed  low-rate  production  of  the 
F-15  prior  to  the  completion  of  developmental  testing.  He 
established  a  policy  of  separate  production  rates.  Under 
this  policy,  concurrency  would  be  accepted  with  low-rate 
production  but  high-rate  production  would  not  commence  until 
the  system  evaluation  was  completed  C 16: 53-60).  Several 
applications  of  prototype  development  have  led  to  some  very 
effective  and  successful  systems  such  as  the  F-16  and  A- 10 
aircraft  (52:110).  The  avionics  and  cannon  subsystems  for 
the  F-15  were  acquired  through  a  prototype  competition 
(36:30). 

As  the  debate  over  the  use  of  concurrency  continued, 
the  ultimate  decision  level  for  acquisition  programs 
progressed  upward  (36:18,22-24).  Recommendations  of  the 
July  1970  Blue  Ribbon  Defense  Panel  resulted  in  the  creation 
of  the  Defense  System  Acquisition  Review  Council  and  the 
policy  to  determine  operational  suitability  of  systems  prior 
to  making  production  decisions.  Another  outgrowth  of  the 
panel's  findings  was  the  creation  of  today's  Air  Force 
flight  test  program  (54-- 27-28).  The  new  flight  test  DT&E/ 
OT&E  concept  calls  for  test  concurrency.  Under  this 
program,  DT&E  and  Initial  OT&E  (IOT&E)  overlap,  allowing 
contractor  and  Air  Force  personnel  to  evaluate  system 
performance,  specification  compliance,  supportabi 1 ity ,  and 
initial  operational  effectiveness  prior  to  a  production 
decision  (1:8-9). 
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Despite  the  changes  to  the  DOD  acquisition  process  in 
the  areas  of  management  and  testing,  the  acquisition  cycle 


continued  to  lengthen  (12:3). 

Acquisition  Today 

In  1977  a  Defense  Science  Board  Task  Force  was  commis¬ 
sioned  to  study  the  increasing  time  cycle  of  acquisition 
(7:iii).  The  Defense  Science  Board  report  released  in  1978 
stated: 

That  the  acquisition  process  has  gone  to  unreason¬ 
able  limits  in  discouraging  concurrency  and  in  over 
emphasizing  advanced  development  prototypes  even 
when  these  add  more  to  program  cost  and  acquisition 
time  than  they  benefit  it  by  reducing  risk  (7.-V). 

In  analyzing  63  acquisition  programs,  the  task  force  con¬ 
cluded  that  there  was  no  clear  correlation  between  concur¬ 
rency  and  poor  quality  systems  (7:49).  The  report  further 
provided  examples  of  highly  concurrent  programs  which 
successfully  met  schedule,  cost,  and  performance  objectives 
(eg.  F-5E,  Polaris,  Minuteman,  Boeing  727)  (7:50).  In  the 
final  analysis,  the  Defense  Science  Board  report  endorsed 
the  use  of  both  concurrency  and  prototyping  in  the  DOD 
acquisition  process.  The  amount  of  concurrency  and/or  the 
decision  to  use  prototypes  should  be  based  on  the  level  of 
technical  risk  and/or  national  urgency  in  ^ach  acquisition 
program  (7:47,54).  Many  of  today's  weapon  system  acquisi¬ 
tion  programs  use  a  combination  of  concurrency  and  proto¬ 
typing.  Some  examples  of  these  combined  programs  are  the 
B- IB,  F- 16  C/D,  LANTIRN,  and  AMRAAM  (5:44;  36:30;  45:7). 


To  complete  this  review  of  concurrency  and  the  DOD 

acquisition  process  changes,  requires  a  mention  of  two 

relatively  recent  events.  The  1981  Acquisition  Improvement 

Program  (AIP),  previously  the  Carlucci  Initiatives,  and 

President  Reagan's  Blue  Ribbon  Commission  on  Defense 

Management,  the  Packard  Commission,  have  made  additional 

changes  and  recommendations  to  correct  problems  in  the 

acquisition  process  and  structure.  Acquisition  problems 

continue  to  be  excessive  time,  cost,  and  the  need  to  improve 

performance  and  readiness  (1:2;  22:63).  Both  the  AIP  and 

Packard  Commission  acknowledged  the  potential  value  of 

concurrency  in  the  acquisition  process-  It  i3  too  early  to 

determine  the  effects  of  the  AIP  and  the  Packard  Commission 

recommendations  on  DOD  acquisition  (1:37-38;  16:72).  One  of 

the  findings  of  the  Packard  Commission  as  summarized  in  the 

June  1986  Air  Force  Magazine  was: 

...that  successful  programs  were  marked  by  devel¬ 
opment  times  of  four  to  five  years-about  half  the 
average.  They  all  share  certain  key  traits: 
short,  clear  lines  of  command;  strict  adherence  to 
program  performance,  cost,  and  schedule  baselines; 
small,  high  quality  staffs;  limited  reporting 
requirements;  good  communication  with  the  end  user; 
extensive  use  of  prototyping  and  operational 
testing  (26:31). 

The  recommendations  of  the  Packard  Commission  will  be 
implemented  in  the  Advanced  Tactical  Fighter  program.  The 
results  of  this  model  program  may  provide  the  answers  to  our 
acquisition  problems  (51=86). 
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Concurrency  Pros  and  Cons 


According  to  Colonel  Bradson,  the  AIP  places  increased 

emphasis  on  concurrent  activities  especially  in  the  areas  of 

DT&E  and  OT&E  (4=173).  The  1978  Defense  Science  Board 

acknowledged  the  benefits  of  concurrency  by  reporting: 

A  certain  amount  of  program  concurrency  can  con¬ 
tribute  to  the  shortening  of  the  acquisition 
process,  with  the  attendant  savings  in  total  acqui¬ 
sition  cost  and  an  increased  return  on  investment  in 
terms  of  the  availability  of  modern  tools.... for  a 
longer  period  of  time  before  obsolescence  (7=46-47). 

In  the  Fleet  Ballistic  Missile  program  concurrency  benefits 
included;  lower  cost,  early  design  maturity,  early  visibil¬ 
ity  of  production  rate  problem,  and  reduced  time  from  system 
conception  to  deployment  (23=74).  If  concurrent  programs 
consistently  produced  these  types  of  benefits,  most  of  the 
DOD  acquisition  problems  would  be  eliminated. 

As  discussed  earlier,  not  everyone  agrees  concurrency 
is  the  solution  to  our  acquisition  problems.  Representative 
Smith,  Co-chairman  of  the  Military  Reform  Caucus  commented, 
"concurrency  isn't  really  needed  unless  we're  in  a  real 
wartime  situation"  (37=46).  At  one  time,  Mr.  Packard 
rejected  the  use  of  concurrency  because  it  produced  cost 
overruns,  took  too  long  to  develop  systems,  and  resulted  in 
low  reliability  (40=3-4).  A  review  of  GAO  reports  also 
shows  recommendations  to  limit  or  avoid  the  use  of  concur¬ 
rency.  GAO  concluded  that  concurrency  increases  program 
risk,  raises  costs,  results  in  lower  performance,  and 
provides  inadequate  test  data  for  decisions  (5=44;  21=4-5). 
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A  1983  study  by  Terence  R.  ST.  Louis  stated  that  prior  to 

1970  some  concurrent  acquisition  programs  failed  because 

OT&E  was  performed  after  the  system  was  in  production. 

Consequently,  operational  deficiencies  were  not  identified 

or  corrected  prior  to  production  C49:13). 

The  debate  over  concurrency 's  use  in  DOD  development 

programs  is  beginning  to  focus  on  a  discussion  of  how  much 

concurrency  is  really  needed.  The  Defense  Science  Board, 

1978,  concluded  that  to  be  successful  the  right  amount  of 

concurrency  should  be  used  based  on  the  level  of  program 

risk  and  the  system  urgency  of  need  C7:51).  Several 

criteria  exist  for  determining  a  program's  success.  For  a 

long  time  the  criteria  was  limited  to  schedule,  cost,  and 

operational  capability.  However,  recently  additional 

considerations  have  been  added  to  the  determination  of 

program  success.  A  1984  memorandum  from  Secretary  of  the 

Air  Force  Verne  Orr  and  Chief  of  Staff  Gen.  Gabriel  stated: 

For  too  long,  the  reliability  and  maintainability 
of  our  weapon  systems  have  been  secondary  consid¬ 
erations  in  the  acquisition  process.  It  is  time  to 
change  this  practice  and  make  reliability  and  main¬ 
tainability  primary  considerations  C29:l). 

Reliability  and  Maintainabi 1 ity 

R&M  issues  have  been  receiving  increasing  management 
attention  for  the  past  few  years.  As  the  cost  and  sophisti 
cation  of  Air  Force  weapon  systems  have  increased,  the  need 
to  improve  system  R&M  has  grown  in  importance  C4:16).  Reli 
ability  is  the  probability  that  an  item  will  perform  a 
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required  function  under  specified  conditions  for  a  specified 

period  of  time.  Also  expressed  as  the  average  time  an  item 

will  perform  a  specified  function  without  failure"  (11:37; 

53 E- 14) .  The  level  of  reliability  obtained  by  a  weapon 

system  determines  the  importance  of  system  maintainability 

(32:25).  AFR  80-14  defines  maintainability  as  "a  measure  of 

the  time  or  maintenance  resource  needed  to  Keep  an  item 

operating  or  restore  it  to  operational  (or  serviceable,  in 

the  case  of  munitions)  status"  (11:35).  According  to  Major 

Hodgson,  DOD  Directive  5000.40  defines  maintainability  as: 

That  ability  of  an  item  to  be  retained  in  or 
restored  to  specified  conditions  when  maintenance  is 
performed  by  personnel  having  specified  skill 
levels,  using  prescribed  procedures  and  resources. . • 
(30:10).. 

The  combination  of  system  reliability  and  maintainability 
produces  system  inherent  availability  (32:25).  Availability 
is  the  "probability  that  an  item  is  in  an  operable  state  at 
a  random  point  in  time  when  used  under  stated  conditions" 

( 53 : E- 14) . 

The  relationship  of  R&M  to  system  availability  and 
combat  capability  makes  R&M  important  concepts  in  today's 
acquisition  environment  (53:E-21).  An  April  30,  1981  Deputy 

Secretary  of  Defense  Memorandum  to  improve  the  acquisition 
process  called  for  designed  in  R&M  as  a  means  to  improve 
system  readiness  (4:62).  General  Mullins  acknowledged 
reliability  as  the  true  measure  of  merit  of  a  weapon  system 
and  the  single  most  limiting  factor  in  terms  of  accomplish- 


merit  of  wartime  taskings  C38:14).  DOD  has  been  involved  in 
reliability  improvement  programs  for  many  years. 

Reliability  History  C39) 

A  paper  titled  "A  Reliability  Chronology"  by  Thomas  A. 
Musson,  C.P.L.  provides  a  look  at  the  emergence  of  relia¬ 
bility  in  system  acquisition  since  the  1950s.  Mr.  Musson 
provides  a  short  evolution  of  three  aspects  of  the  DOD 
reliability  program.  He  focuses  on  the  changing  objective 
of  reliability  improvements,  the  management  reviews  and 
directives,  and  the  search  for  a  solution  to  reliability 
problems  over  the  past  three  decades. 

As  pointed  out  in  the  article,  the  objectives  of  relia¬ 
bility  activities  have  always  been  focused  on  a  bigger  issue 
facing  the  Department  of  Defense.  The  changes  in  improved 
reliability  objectives  resulted  from  either  attainment  of 
the  objective  or  a  change  in  the  environment.  Beginning  in 
the  1950s  emphasis  was  on  increasing  the  operational  time  of 
equipment  to  keep  it  working.  Next  the  focus  moved  to 
insure  improved  mission  reliability.  A  tightening  defense 
budget  in  the  1970s  caused  a  shift  to  the  cost  reduction 
benefits  of  improved  reliability.  This  fostered  development 
of  baselines  and  life  cycle  cost  and  logistic  support  cost 
analysis.  A  preoccupation  with  reducing  costs  caused  a 
shift  away  from  the  ultimate  goal  of  a  weapon  system  which 
is  to  provide  combat  capability.  In  the  past  ten  years,  DOD 
and  the  Air  Force  have  managed  reliability  improvement 
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programs  to  balance  system  cost  and  readiness.  Along  with 
these  objectives,  improved  reliability  will  reduce  the 
quantity  and  skill  levels  of  personnel  required  to  maintain 
current  and  future  weapon  systems. 

Mr.  Musson  identifies  six  significant  events  in  the 
shaping  of  reliability  programs.  In  1952,  the  Advisory 
Group  on  Reliability  of  Electronic  Equipment  CAGREE)  was 
established.  AGREE  was  comprised  of  nine  task  groups  which 
worked  to  improve  electronic  equipment  reliability  by 
developing  programs  in  the  areas  of  numerical  reliability 
requirements,  tests,  design  procedures,  components,  procure¬ 
ment,  packaging  and  transportation,  storage,  and  operation 
and  maintenance.  One  requirement  for  improving  reliability, 
identified  by  AGREE,  resulted  in  the  formation  of  the  Ad  Hoc 
Study  Group  on  Parts  Specification  Management  for  Reliabil¬ 
ity.  This  group's  report  resulted  in  establishment  of 
reliability  military  specifications  for  piece  parts  such  as 
transistors  and  diodes.  In  the  1970s  the  Electronics-X 
Study  by  the  Institute  for  Defense  Analysis  and  the  Joint 
Logistics  Commanders  Electronic  Systems  Reliability  workshop 
provided  the  impetus  towards  the  use  of  warranties  and 

incentives  to  reduce  cost  and  improve  reliability.  The  most 

* 

recent  events  identified  in  this  article  were  the  publishing 
of  the  DOD  Directive  on  Reliability  and  Maintainability, 
5000.40,  and  the  Defense  Acquisiton  Improvement  Program. 
These  actions  were  taken  to  increase  the  overall  efficiency 
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and  effectiveness  of  the  reliability  program  and  to  focus 
attention  on  reliability  as  a  design  factor. 

One  of  the  reasons  for  so  many  management  evaluations 
of  DOD  reliability  activities  was  the  search  for  a  simple 
solution  to  insure  that  the  desired  level  of  reliability  was 
obtained  in  the  weapon  systems.  Many  factors  have  resulted 
in  DOD  not  reaching  its  reliability  goals.  These  include 
the  resources  and  commitment  made  towards  the  reliability 
effort,  and  the  complexity  and  technical  challenge 
associated  with  some  particular  items.  Cost  and  the  unwill¬ 
ingness  to  trade  some  system  capability  for  reliability 
gains  has  also  limited  the  overall  results.  Initially,  the 
use  of  statistics  and  quantitative  measures  were  considered 
the  answer  to  improved  reliability.  From  this  ability  to 
define  and  measure  reliability,  DOD  focused  on  piece  part 
specifications  as  the  key  to  better  systems.  Testing  and 
demonstration  of  specification  compliance  by  the  contractor 
also  failed  to  produce  the  desired  field  reliability  because 
the  test  environment  did  not  duplicate  the  operational 
environment.  DOD  next  focused  on  the  use  of  Reliability 
Improvement  Warranties  CRIW).  RIW  has  in  some  cases 
improved  system  reliability.  The  current  solution  is  to 
focus  on  reliability  as  an  inherent  characteristic  of 
design.  Each  of  the  identified  solutions  provide  a  means  of 
insuring  improved  system  reliability.  None  can  stand  alone 


and  the  proper  mix  during  system  acquisition  and  system  life 
cycle  will  provide  the  highest  reliability  possible. 

Improving  Rel iabl 1 ity 

According  to  Captain  Demarchi,  Igor  Bazovsky  stated  the 

best  and  cheapest  reliability  improvements  are  obtained 

during  system  design  through  tradeoffs  in  system  performance 

(9=12).  Reliability  improvements  will  increase  the  price  of 

new  systems  by  10  to  15  percent  (44:44).  However,  this 

investment  provides  a  force  multiplier  through  increased 

sortie  production  and  logistics  sustainability  which 

improves  system  availability  and  combat  effectiveness 

(44:45;  53:E-14, E-21).  By  increasing  system  R&M,  the  Air 

Force  can  reduce  the  quantity  and  cost  of  support  personnel, 

equipment,  and  spare  parts  (30:11;  53:E-13). 

R&M  improvements  begin  early  in  the  acquisition  cycle 

with  better  R&M  specifications  stated  as  contractual  goals 

(53:E-13).  In  1981  a  design  engineering  emphasis  panel 

identified  the  most  cost  effective  reliability  tasks  as  parts 

derating,  parts  selection  and  control,  failure  analysis  and 

corrective  action,  parts  screening,  and  burn-in  (15=25). 

Derating  is  the  practice  of  reducing  the  elec¬ 
trical,  mechanical,  or  environmental  operating 
stresses  below  the  maximum  levels  the  part  is 
capable  of  sustaining  ...  resulting  in  an  increased 
part  lifetime  (15:28). 

Burn-in  is  submitting  deliverable  end  items  to  a 
short  te3t,  at  the  highest  practical  level  of 
assembly,  to  disclose  weak  parts  and  manufacturing 
defects  for  correction  prior  to  delivery  (50:143). 
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Several  DOD  programs  have  been  credited  with  achieving 


successful  R&M  standards.  R&M  improvements  were  obtained 
through  design  and  system  modifications  after  production 
start.  The  F-15  and  F-16  aircraft  are  examples  of  improved 
R&M  during  the  design  process  when  compared  to  the  F-4.  The 
F-15  requires  one  third  fewer  manhours  per  flying  hour  and 
the  F-16  requires  only  half  the  manhours  per  flying  hour  of 
the  F-4  (17:10-11).  The  Army's  Black  Hawk  helicopter  proved 
that  performance  and  reliability  can  be  coproduced  (34:39). 
According  to  General  Russ,  R&M  modifications  to  the  F-15 
fleet  since  1974  resulted  in  a  50  percent  increase  in 
missions  per  aircraft  per  month  in  1985  (17:10).  Other 
systems  acknowledged  as  current  R&M  successes  are  the  Navy 's 
F-18  aircraft,  the  Army's  M- 1  tank  and  Firefinder  Radar 
system,  and  the  Air  Force's  F-16  APG-66  radar  (34:39; 

53 : E- 12, E-18). 

Improved  weapon  system  reliability  will  provide 
significant  returns  in  supportabi 1 ity .  These  improvements 
can  be  obtained  in  three  ways:  early  design,  testing  and 
redesign  after  proauction  start,  and  through  system  maturity 
(34:39).  By  doubling  the  reliability  of  the  F-16  avionics 
and  engine  subsystems,  spare  parts  costs  and  manpower 
requirements  could  be  reduced  by  45  and  40, percent,  respec¬ 
tively  (27:83).  Another  example  of  increased  savings  from 
improved  R&M  was  provided  by  General  Mullins.  Increasing  a 
system  reliability  25  percent  from  a  mean  time  between 


25 


failures  (MTBF)  of  500  hours  to  625  hours,  results  in  a  40 
percent  reduction  in  spares  requirements  while  maintaining 
the  same  aircraft  availability  (38:16). 

In  1972  David  Packard  identified  one  way  to  obtain 
system  reliability  build  it,  test  it,  and  fix  things  that  go 
wrong;  then  repeat  until  desired  reliability  is  achieved 
(4: 5-6) .  Today's  test  and  evaluation  policy  is  designed  to 
discover  system  capabilities  and  limitations  before  a  system 
is  produced  and  deployed  (49:11).  The  Acquisition  Improve¬ 
ment  Program  established  initiatives  to  provide  adequate 
front  end  funding  for  test  hardware  to  shorten  acquisition 
time  without  increasing  risks  (18:6).  These  initiatives 
would  allow  for  concurrent  development  and  testing  phases 
(3:11;  18:6). 

...if  high  confidence  is  to  be  placed  on  relia¬ 
bility  and  support  goals  to  be  achieved  in  fielded 
defense  systems,  iterative  design  and  testing  must 
be  conducted  before  full  rate  production  (47:9). 

An  iterative  test,  analyze,  and  fix  phase  (TAAF)  is  an 

important  element  in  producing  reliable  systems. 

TAAF  is  a  technique  for  reliability  development  and 
growth  testing  that  requires  that  a  series  of  tests 
be  conducted,  problems  identified  and  analyzed,  and 
corrective  actions  taken.  TAAF  provides  a  means  to 
accelerate  design  and  reliability  maturity  by 
correcting  Identified  design  performance  and 
reliability  problems  (41:12-13). 

However,  managers  must  be  cautious  in  accepting  design  fixes 
without  verification.  Fixes  may  result  in  new  system 
problems  due  to  new  failure  types  and  other  component  inter¬ 
actions  (32:26). 
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Problems  in  Reaching  R&M  Goals 

Efforts  to  reduce  costs  and  improve  system  reliability, 
maintainability,  and  availability  have  not  been  as  successful 
as  desired  (8: A- 13).  The  available  literature  provides 
numerous  interrelated  causes  for  the  failure  of  some  systems 
to  reach  R&M  goals.  During  the  system  acquisition  cycle, 
decisions  which  affect  R&M  may  be  driven  by  funding  and 
schedule  constraints  resulting  in  reduced  R&M  (15: 27).  A 
weapon  system's  operational  performance  receives  the  atten¬ 
tion  of  management  and  Congress.  Good  performance  prevents 
funding  cuts  (34:38-39).  The  competition  for  limited 
resources  results  in  intense  congressional  focus  on  acquisi¬ 
tion  programs.  Congress  has  been  reluctant  to  support  the 
increased  upfront  costs  required  for  R&M  improvements  (6:9: 
34:39). 

...  deferring  the  near  term  costs  of  R&M  testing 
and  design  improvements,  due  to  program  cost  and 
schedule  constraints,  can  result  in  significantly 
higher  support  costs  over  the  extended  system  life 
cycie  (30 :  11) . 

The  USAF  R&M  Action  Plan  Development  Team  reported  that 
attempts  to  reduce  the  acquisition  cycle  are  often  cited  as 
the  cause  of  reduced  R&M  so  that  cost,  schedule,  and 
performance  improvements  can  be  realized  (53:20).  Steps 
taken  to  shorten  acquisition  times  may  involve  system 
production  prior  to  completing  development  model  testing. 

This  results  in  the  use  of  concurrency  (47:9).  A  1980  study 
by  the  Vought  Corporation  found  that  R&M  advances  were  some- 
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times  off-set  by  the  addition  of  more  equipment  to  improve 
weapon  system  performance  and  capability  C 33: 4-5). 

The  determination  of  a  system's  R&M  is  accomplished 
through  data  collection  during  testing  and  again  in  field 
operation.  In  1977  Major  Richard  Rose  discussed  the  relia¬ 
bility  problem  with  several  former  Air  Force  program 
managers.  Many  of  these  managers  did  not  feel  there  was  a 
reliability  problem  since  most  programs  achieved  contractual 
reliability  goals  (43:9-10).  A  1979  GAO  report  which 
covered  reviews  of  21  major  DOD  systems  indicated  a  relia¬ 
bility  problem  in  acquisition.  GAO  found  a  few  cases  where 
systems  either  failed  to  reach  reliability  goals  or  showed  a 
potential  for  reliability  problems  during  system  testing 
( 21 : 3-4)  . 

R&M  goals  are  usually  expressed  as  measures  of  central 
tendency  MTBF  for  reliability  and  mean  time  to  repair  CMTTR) 
for  maintainability  (43:41;  53:E-11).  These  R&M  measures 
are  tracked  throughout  a  system's  acquisition  cycle- 
Several  different  test  phases  are  used  to  develop  predic¬ 
tions  of  a  system's  field  R&M. 

DT&E  is  that  testing  and  evaluation  used  to  measure 
progress,  verify  accomplishment  of  development 
objectives,  and  to  determine  if  theories,  tech¬ 
niques,  and  material  are  practible;  and  if  systems 
or  items  under  development  are  technically  sound, 
reliable,  safe,  and  satisfy  specifications  (11=j4). 

OT&E  is  testing  and  evaluation  conducted  in  as 
realistic  an  operational  environment  as  possible  to 
estimate  the  prospective  system's  military  utility, 
operational  effectiveness,  and  operational  suit¬ 
ability.  In  addition  OT&E  provides  information  on 
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organization,  personnel  requirements,  doctrine,  and 
tactics.  Also,  it  may  provide  data  to  support  or 
verify  material  in  operating  instructions,  publica¬ 
tions,  and  handbooks  (11=36). 

IOT&E  is  the  first  phase  of  operational  test  and 
evaluation  conducted  on  preproduction  items,  proto¬ 
types,  or  pilot  production  items  and  normally 
completed  prior  to  the  first  major  production 
decision  (11:35). 

Testing  and  evaluation  (T&E)  are  key  elements  in  determining 
the  capabilities  and  limitations  of  systems.  Despite  the 
importance  of  T&E,  its  accompl ishment  is  affected  by  limits 
of  time,  money,  and  resources  resulting  in  the  use  of 
concurrent  DT&E  and  IOT&E  (49:33).  Some  critics  of  current 
acquisition  programs  feel  that  OT&E  does  not  catch  problems 
in  time  to  keep  them  from  going  into  production  and  at  times 
data  is  fudged  to  prevent  the  surfacing  of  problems  (5:41). 
Several  studies  have  shown  that  "predicted  reliability  does 
not  correlate  well  with  field  experience"  (14:56;  15=26). 

A  1977  study  by  Hughes  Aircraft  Company  which  examined 
16  different  types  of  avionics  systems  provided  some  expla¬ 
nations  for  the  variances  between  predicted  and  field  relia¬ 
bility.  The  findings  showed  that  field  reliability  was 
sometimes  dependent  on  the  type  of  aircraft.  Like  equipment 
installed  in  subsonic  bombers  and  transports  exhibited  2  to 
4  times  higher  reliability  than  when  installed  in  fighters 
and  trainers.  Hughes  personnel  also  found  as  much  as  a  5=' 
difference  in  reliability  based  on  weapon  system  location. 
This  variance  was  attributed  to  differences  in  maintenance 
practices,  manpower  skill  levels,  and  the  geographic/ 
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climatic  environment  (14:56).  Additional  hindrances  to 

duplicating  T&E  R&M  results,  experienced  by  user  command 

personnel,  are  problems  in  receiving  test  equipment  and 

spare  parts  delayed  due  to  system  design  changes  C35: 23-24; 

54:61).  Other  factors  which  contribute  to  the  poor  accuracy 

of  R&M  predictions  can  be  found  in  the  T&E  phase.  They  are: 

The  use  of  contractor  personnel  to  maintain 
systems.  Contractor  personnel  are  usually  more 
skilled  and  experienced  than  user  command  techni¬ 
cians  ( 19: 32 ;  46:834).  Inadequate  test  and  support 
equipment  used  during  T&E.  The  use  of  contractor 
equipment  not  available  to  field  units  (20:4,23; 

46:834).  Test  articles  were  not  representative  of 
production  article  (20:10;  46:834).  The  establish¬ 
ment  of  a  logistics  system  dedicated  to  testing 
which  provided  direct  access  to  manufacturers  for 
spare  parts  and  expedited  shipments  (20:23). 

The  link  between  reliability  and  maintainability  causes  the 
problem  of  poor  reliability  predictions  to  be  magnified  in 
the  field.  Reliability  data  produced  during  T&E  is  used  for 
maintainability  predictions,  spares  provisioning,  support 
equipment  utilization,  and  manpower  requirements  (43:85).  A 
RAND  study  of  the  A-7D  aircraft  demonstrated  a  significant 
difference  between  predicted  and  operational  MTBFs.  Five 
major  avionic  systems  were  evaluated  with  the  radar  showing 
the  largest  disparity,  250  hour  predicted  MTBF  compared  to 
an  actual  25  hour  MTBF  (43:10-13,53-54).  A  review  of  the 
F-15  aircraft  program  in  the  early  operational  stage 
provided  data  showing  poor  maintainability  predictions  and 
that  low  reliable  systems  consumed  high  levels  of  mainten¬ 
ance  time  (43:20-21). 
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Anthony  J.  Feduccia,  Systems  Reliability  and  Engineer¬ 
ing  Branch  Chief,  listed  six  reasons  for  the  difference 
between  predicted  and  achieved  reliability:  false  removals, 
definition  of  failure,  maintenance  induced  failures, 
environment,  configuration  changes,  and  spare  parts  (15:26) 
A  major  contributor  to  the  conflicting  results  of  reliabil¬ 
ity  analysis  is  based  on  the  question  of  what  constitutes  a 
failure  (32:27).  During  the  system  acquisition  cycle  repre 
sentatives  from  the  System  Program  Office  (SPO),  DT&E  and 
OT&E  test  team,  and  sometimes  the  contractor  form  a  Joint 
Reliability  and  Maintainability  Evaluation  Team  (JRMET). 

The  purpose  of  JRMET  is  to  collect,  analyze,  and  categorize 
R&M  data  during  T&E  (10:4).  R&M  measures  are  influenced  by 
the  team's  definition  of  what  failures  will  be  counted  in 
the  R&M  calculations.  Several  important  definitional 
guidelines  exist  to  make  this  determination. 

Non-relevant  failures:  only  those  failures  that  are 
caused  by  a  condition  external  to  the  equipment 
under  test  that  is  not  encountered  in  field 
service. 

Relevant  failures:  includes  all  failures  incurred 
during  test  that  can  be  expected  to  occur  in  the 
f ield. 

Relevant  failures  are  further  subdivided. 

Chargeable  failures:  those  relevant  failures 
incurred  during  test  which  are  caused  by  any  of  the 
goods  or  services  provided  by  a  given  contractor. 

Non-chargeable  failures:  those  relevant  failures 
incurred  during  test  which  are  caused  by  and  are 
dependent  upon  a  condition  previously  stipulated  as 
not  within  the  responsibility  of  a  given  contractor 
(50:131). 
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Only  chargeable  failures  as  determined  by  JRMET  are  used  to 
make  R&M  calculations.  This  close  scrutiny  of  failures 
during  T&E  is  not  duplicated  in  the  field.  This  results  in 
inconsistent  R&M  measurements.  While  T&E  requirements  do 
not  count  some  failures,  these  non-chargeable  failures  have 
a  significant  impact  on  manpower,  support  and  test  equipment 
requirements,  and  spares  provisioning.  Examination  of 
actual  data  from  a  fielded  avionic  system  provides  an 
insight  into  problems  from  failure  definition  differences. 
The  contractor's  reliability  prediction  was  a  150  hour  MTBF ; 
however,  the  field  showed  an  average  of  only  75  hours.  The 
differences  in  MTBF  values  was  due  to  the  inclusion  of 
failures  by  the  field  which  included  induced  failures  and 
maintenance  actions  such  as;  could  not  duplicate,  retest-OK, 
bench  check  serviceable  and  adjustments.  While  these  field 
counted  failures  are  not  considered  chargeable  failures 
during  T&E,  they  do  have  a  significant  impac';  on  maintenance 
of  the  system  (15:26;  41:14).  Besides  the  impact  of  MTBF 
differences  on  a  fielded  system's  maintainability,  the  use 
of  MTTR  as  a  measure  of  maintainability  presents  problems. 
MTTR  calculations  do  not  include  time  to  locate  spares  and 
test  equipment,  or  downtime  due  to  human  or  software  errors. 
These  occurrences  produce  as  many  problems  for  maintenance 
personnel  as  hardware  failures  (15:25).  The  continued  slow 
progression  of  R&M  improvements  in  Air  Force  weapon  systems 
led  to  the  establishment  of  the  Air  Force  R&M  2000  program. 


R&M  2000 


The  Air  Force  R&M  2000  Program  was  designed  to  mobilize 
senior  level  management  interest  in  improving  weapon  system 
R&M  and  to  institutionalize  R&M  concepts  (46=44;  55=51). 

The  basis  for  development  of  R&M  2000  was  the  perceived 
Soviet  threat  to  European  depots,  airlift  limitations,  and 
the  future  expected  shortfall  in  maintenance  personnel 
(27:81,83;  44=44-45).  This  current  R&M  improvement  program 
is  also  directed  at  DOD  contractors  and  has  the  management 
support  needed  to  minimize  the  trade-off  of  R&M  requirements 
for  cost,  schedule,  and  performance  (24=49;  53=E-11).  The 
R&M  2000  Program  involves  both  development  and  fielded 
systems  and  is  supported  by  all  major  commands.  The  Air 
Force  is  serious  about  improving  weapon  system  R&M  as 
demonstrated  by  programs  being  delayed  or  cancelled  for 
failure  to  meet  R&M  goals/requirements  (25=16;  27=83,85). 

One  Air  Force  source  told  Defense  Electronics  "this  time, 
we're  not  just  asking  contractors  to  beef  up  reliability," 
..."this  time,  it's  an  ultimatum"  (27=81).  By  increasing 
system  reliability  and  maintainability  the  Air  Force  will 
meet  the  goals  of  the  R&M  2000  Action  Plan: 

Increase  warfighting  capability. 

Increase  survivability  of  the  combat*  support 

structure. 

Decrease  mobility  requirements  per  deploying  unit. 

Decrease  manpower  requirements  per  unit  of  output. 

Decrease  costs  (41=1). 
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Summary 


This  chapter  provided  a  review  of  the  use  of  concur¬ 
rency  in  the  DOD  acquisition  process.  The  main  reasons 
presented  for  concurrent  acquisition  programs  are  to  reduce 
costs  and  development  time.  Some  negative  aspects  identi¬ 
fied  with  concurrency's  use  are  that  it  results  in  lower 
performance,  provides  inadequate  test  data  for  program 
decisions,  and  reduces  reliability.  The  current  acquisition 
philosophy  accepts  the  use  of  concurrency  based  on  system 
need  and  program  risk.  Some  level  of  concurrency  appears  to 
exist  in  all  programs. 

Also  during  this  literature  review,  the  researcher 
covered  the  areas  of  system  R&M.  Reliability  and  maintain¬ 
ability  were  defined.  The  increasing  costs  of  new  weapon 
systems  has  made  R&M  important  aspects  of  system  acquisi¬ 
tion.  While  R&M  has  received  increased  management  atten¬ 
tion,  the  R&M  measures  of  MTBF  and  MTTR  have  failed  to  reach 
desired  field  levels.  This  results  in  reduced  weapon  system 
capability  and  availability.  A  few  of  the  reasons  for  R&M 
problems  can  be  found  in  the  structure  of  T&E  phases  and  in 
the  calculations  of  R&M  measures.  The  next  chapter  will 
provide  the  methodology  used  in  this  thesis  to  determine 

t 

concurrency 's  impact  on  R&M  and  acquisition  personnel  views 
on  the  use  of  concurrency. 


34 


III.  Research  Methodology 


Data  Collection 

A  literature  review  was  conducted  to  find  historical 
examples  of  the  use  of  concurrency  in  system  acquisitions. 
During  this  review,  the  researcher  concentrated  on  investi¬ 
gating  the  reasons  that  concurrency  ha3  been  identified  as 
having  both  a  positive  and  negative  impact  on  the  acquisi¬ 
tion  process.  The  researcher  examined  the  literature  and 
obtained  a  perspective  on  weapon  system  R&M.  This  review 
focused  on  reporting  the  benefits  and  hindrances  to  R&M 
improvements.  The  researcher  found  several  references  which 
identified  some  problems  with  the  quantitative  measurements 
of  R&M  developed  during  T&E  in  accurately  predicting  system 
field  performance. 

A  qualitative  determination  of  the  impact  of  concur¬ 
rency  on  acquisition  and  system  R&M  was  obtained  through 
interviews  of  managers  actively  Involved  in  the  acquisition 
process.  The  personnel  interviewed  were  assigned  to  the 
following  organizations/offices:  AFSC/System  Program  Office 
CSPO),  AFLC/Deputy  Program  Manager  for  Logistics  CDPML),  and 
AFOTEC/Operational  Test  Team.  The  researcher  chose  these 
organizations/of f ice3  to  obtain  a  comparative  .data  base  on 
concurrency.  Based  on  the  manager's  position  and  experience 
with  concurrency's  affect  on  acquisition  and  R&M,  different 
responses  could  result  when  compared  to  other  program  areas. 
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Procedure 


To  determine  the  programs  in  which  to  conduct  inter¬ 
views,  the  researcher  spoke  with  the  Deputies  in  the  Acqui¬ 
sition  Divisions  located  at  Wright-Patterson  AFB.  The 
person  contacted  in  each  Acquisition  Division  provided  a 
number  of  programs  which  would  meet  the  program  selection 
criteria.  Acquisition  programs  selected  for  inclusion  in 
this  thesis  used  concurrency  and  were  close  to  or  past  the 
production  decision.  After  a  program  was  identified  to  be 
used  in  this  study,  the  personnel  selected  for  the  inter¬ 
views  were  determined  by  the  cluster  sample  technique.  In 
the  cluster  sample  technique,  "...the  entire  population  is 
divided  into  groups  of  elements  and  some  of  the  groups  for 
study  are  randomly  selected"  (13:312).  For  this  thesis  the 
population,  all  acquisition  personnel,  was  divided  into 
groups,  a  specific  weapon  system  program.  After  a  program 
was  identified  as  meeting  the  thesis  selection  criteria,  the 
interviewees  were  chosen.  Five  acquisition  programs  were 
used  in  this  thesis  and  personnel  assigned  to  the  SPO,  DPML, 
and  AFOTEC  areas  in  each  program  were  interviewed.  The 
listing  of  the  interviewees  can  be  found  in  Appendix  A. 

Interviews  were  chosen  as  the  means  to  collect  the  data 
for  this  thesis  because  the  use  and  definition  of  concur¬ 
rency  varies.  Also,  the  definition  of  R&M  and  its  quanti¬ 
tative  measures  vary.  During  interviews  the  researcher  was 
able  to  account  for  these  differences.  Interviews  of  SPO 
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and  DPML  personnel  were  arranged  and  conducted  in  person  at 
Wright-Patterson  AFB.  These  interviews  were  taped  to  insure 
accuracy.  However,  interviews  of  AFOTEC  personnel  were 
conducted  over  the  phone  and  required  manual  recording  of 
the  responses.  When  appropriate,  a  transcript  of  the 
interviews  has  been  provided  in  Appendixes  B  to  H.  In  some 
cases  the  interviewees  were  not  able  to  respond  to  all  the 
interview  questions.  To  ensure  non-attribution,  the  inter¬ 
viewees  were  assigned  a  random  number.  This  number  was  used 
to  consistently  identify  a  particular  individual's  responses 
without  identifying  the  individual.  Also,  all  references  to 
the  specific  program  were  removed  from  the  transcripts. 

Interview  Questions  Development 

The  interview  questions  were  designed  to  answer  the 
investigative  questions  in  terms  of  factors  which  influence 
the  acquisition  process,  specifically  in  the  areas  of  R&M- 
The  factors  used  were  determined  through  a  review  of  the 
literature  (15:25-26;  20:4,10,23;  21:8,10).  Interview 
questions  7  through  14  are  based  on  these  factors.  The 
interview  questions  are  listed  below: 

1.  What  is  your  general  assessment  of  the  use  of 
concurrency  in  acquisition? 

2.  What  phase  of  the  acquisition  process  'is  your  program 
currently  in? 

3.  What  was  the  original  risk  assessment  of  your  program? 

4.  Was  the  use  of  concurrency  planned  from  the  beginning 
of  your  program? 
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5.  If  concurrency  was  not  originally  planned,  why  was  it 
later  incorporated  and  in  what  phase  of  acquisition? 

6.  What  phases  of  your  program  were  concurrent? 

7.  Who  was  responsible  for  the  maintenance  of  your  system 
during  DT&E  and  IOT&E  and  at  what  level?  Corganiza- 
tional,  intermediate,  depot)  If  the  contractor,  does 
this  bias  the  R&M  data? 

8.  What  was  the  status  of  the  system  unique  support  equip¬ 
ment  during  DT&E  and  IOT&E? 

9.  What  was  the  status  of  the  system  unique  test  equip¬ 
ment  during  DT&E  and  IOT&E? 

10.  What  was  the  status  of  technical  data  during  DT&E  and 
IOT&E? 

11.  Did  concurrency  cause  a  delay  in  DT&E  to  complete 
IOT&E  in  order  to  meet  a  production  decision  date? 

What  problems  did  this  cause? 

12.  Did  concurrency  result  in  a  design  freeze  earlier  than 
planned? 

13.  Were  there  open  engineering  change  proposals  (ECP) 
dealing  with  R&M  issues  at  the  time  of  IOT&E?  Will 
these  changes  result  in  a  significant  design  differ¬ 
ence  between  the  tested  and  production  articles? 

14.  Did  concurrency  reduce  your  ability  to  use  a  test, 
analyze,  and  fix  procedure  to  identify  potential  reli¬ 
ability  problems? 

15.  How  does  concurrency  benefit  R&M? 

16.  What  are  the  primary  benefits  of  a  concurrent  program? 

17.  What  problems  are  associated  with  a  concurrent  program? 

18.  What  needs  to  be  done  to  improve  system  R&M  during 
acquisition? 

« 

19.  What  do  you  feel  accounts  for  the  differences  in  R&M 
measures  between  DT&E,  IOT&E,  and  field  results? 
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Data  Analysis 


The  initial  results  of  this  research  provided  a  back¬ 
ground  of  concurrency  and  identified  the  reported  benefits 
and  problems.  The  researcher  compiled  published  data  on  the 
adequacy  of  R&M  test  data  in  predicting  system  field  data. 
Answers  to  Questions  2  through  6  were  used  to  compare  the 
programs  in  this  study  and  are  not  included  in  an  appendix 
but  are  summarized  in  the  findings  chapter.  Managers' 
opinions  on  the  use  of  concurrency  were  documented  through 
the  remaining  questions.  A  statistical  analysis  was 
conducted  on  the  responses  to  interview  questions  7  through 
14  using  descriptive  statistics  based  on  the  answers  to  the 
hypotheses  listed  in  the  next  section.  The  data  was  evalu¬ 
ated  in  terms  of  frequency  and  count  of  the  individual 
manager's  responses. 

Hypotheses 

A  hypothesis  was  developed  for  each  interview  question 
C 7- 14)  to  ascertain  the  manager's  opinion  of  factors  which 
may  impact  reliability  and  maintainability  when  comparing 
DT&E/IOT&E  results  to  the  operational  field  reported  R&M. 
Each  interviewed  individual 's  answer  was  recorded.  However, 
some  interviewees  could  not  respond  to  all  the  questions 
presented.  The  hypotheses  developed  for  this  evaluation 
are: 

1.  Use  of  contractor  personnel  for  system  maintenance 

reduces  the  accuracy  of  system  R&M  measures.  CQues.  7) 


39 


2.  Lack  of  system  unique  support  equipment  during  testing 
reduces  the  accuracy  of  system  R&M  measures.  CQues.  8) 

3.  Lack  of  system  unique  test  equipment  during  testing 
reduces  the  accuracy  of  system  R&M  measures.  CQues.  9) 

4.  The  lack  of  system  technical  data  during  DT&E  and 
IOT&E  reduces  the  accuracy  of  system  R&M  measures. 

CQues.  10) 

5.  Incomplete  DT&E  and  IOT&E  testing  prior  to  the  produc¬ 
tion  decision  reduces  the  accuracy  of  system  R&M 
measures.  CQues.  11) 

6.  Early  system  design  freeze  for  IOT&E  will  reduce  the 
accuracy  of  system  R&M  measures.  CQues.  12) 

7.  Open/ untested  ECPs  at  the  time  of  production  decision 
reduces  the  accuracy  of  system  R&M  measures.  CQues.  13) 

8.  An  incomplete  test,  analyze,  and  fix  program  during 
development  reduces  the  accuracy  of  system  R&M 
measures.  CQues.  14) 

An  example  of  the  method  used  to  categorize  the  inter¬ 
view  data  follows  in  Table  1. 


Table  1 

Hypothesis  7  Interview  Question  13 
Example 


Hypothesis:  Open/ untested  ECPs  for  reliability  improvements 

at  the  time  of  the  production  decision  reduces  the  accuracy 
of  system  R&M  measures. 


Did  open/ untested  ECP  reliability 
improvements  exist  at  the  time  of 
the  production  decision? 

YES  NO 

Reduces  accuracy  of  YES  X 

system  R&M  measures 

NO 
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In  this  example  the  interviewee  stated  that  open  ECPs 
did  exist  at  the  time  of  the  production  decision  and  that 
this  did/or  will  result  in  lower  field  R&M  measures. 

Limitations 

The  factors  identified  as  having  an  impact  on  system 
reliability  and  maintainability  cannot  be  empirically 
demonstrated. 

Only  Air  Force  acquisition  programs  which  had  already 
experienced  a  production  decision  were  used  to  select 
personnel  to  be  interviewed.  This  selection  criteria 
resulted  in  a  small  data  base  with  limited  generalization 
capabi 1 ity . 

Summary 

Part  of  the  data  base  for  this  thesis  was  developed 
through  a  literature  review  of  concurrency  and  R&M  material. 
Data  collection  was  directed  at  determining  the  relationship 
between  concurrency  and  weapon  system  R&M-  The  researcher 
also  extracted  data  from  the  literature  which  provided  a  set 
of  factors  identified  as  impacting  weapon  system  R&M.  The 
remaining  data  base  was  obtained  through  interviews  with 
acquisition  personnel.  These  interviews  provided  the  infor¬ 
mation  used  to  determine  the  impact  of  concurrency  on  acqui¬ 
sition.  Interview  questions  were  also  developed  to  deter¬ 
mine  the  manager’s  opinion  on  whether  or  not  the  factors 
which  may  affect  R&M  had  an  impact  on  their  program. 
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IV .  Findings 


Introduction 

This  research  effort  used  two  methods  to  develop  a  data 
base.  The  data  collection  was  initiated  with  a  literature 
review  to  obtain  information  on  concurrency's  use  in  weapon 
system  acquisition.  This  information  was  then  used  to 
develop  interview  questions  for  the  purpose  of  obtaining  the 
opinions  of  managers  involved  in  the  acquisition  process. 
This  chapter  provides  findings  to  answer  the  thesis  research 
questions  presented  in  Chapter  I  and  is  divided  into  two 
sections  literature  review  findings  and  interview  findings. 

Literature  Review  Findings 

Weapon  System  Programs  Successful ly  Developed  Using 
Concurrency-  The  Department  of  Defense  began  using  concur¬ 
rency  with  the  advent  of  the  Ballistic  Missile  Program. 

This  highly  concurrent  program  was  initiated  to  develop  an 
ICBM  before  the  Russians.  The  Ballistic  Missile  Program  was 
the  number  one  national  priority  during  its  acquisition  and 
successfully  developed  three  versions;  Atlas,  Titan,  and 
Thor.  More  recently  developed  missile  programs  classified 
successful  by  the  1977  Defense  Science  Board  Task  Force  were 
the  Polaris  and  Minuteman.  The  Task  Force  also  identified 
the  F-5E  aircraft  as  a  highly  concurrent  program  which  met 
cost,  schedule,  and  performance  objectives.  The  U-2  and 
SR-71  aircraft  developed  by  Lockheed  have  proven  to  be  very 
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effective  weapon  systems.  Also  two  of  the  Air  Force's 
modern  fighters  the  F-15  and  F-16  C/D  can  be  classified  as 
successful  programs  which  used  some  level  of  concurrency. 
While  these  weapon  systems  prove  that  concurrency's  use  in 
acquisition  can  produce  systems  which  meet  military  needs, 
concurrency  has  not  been  totally  accepted. 

Concurrency  Has  Been  Periodical ly  Accepted  and 
Rej  ected.  Over  the  years  the  results  of  concurrent  acquisi¬ 
tion  programs  have  ranged  from  unqualified  success  to  total 
failure.  Following  the  development  of  the  ICBMs  and  the  U-2 
and  SR-71  aircraft,  concurrency  was  an  accepted  acquisition 
process.  These  programs  were  acquired  using  concurrency  as 
a  management  philosophy  and  there  are  several  reasons  which 
account  for  their  success.  Concurrency  was  planned  from  the 
beginning  of  the  programs.  Program  management  was  given 
almost  complete  autonomy  with  the  required  overall  respon¬ 
sibility  and  authority  for  all  aspects  of  the  program.  This 
enabled  management  decisions  to  be  made  at  the  lowest  ievel 
possible.  The  program  staff  consisted  of  a  smail  group  of 
highly  qualified  and  skilled  personnel.  To  overcome  the 
high  risk  of  the  new  and  advanced  technology  required  to 
produce  the  ICBM,  competition  was  extensively  used  during 
development.  The  ICBM  program  received  support  from  the 
highest  levels  of  the  military  and  civilian  leadership  as  a 
national  priority  and  did  not  appear  to  have  any  funding 
problems.  Because  there  was  very  limited  access  to  the 
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project  and  its  progress,  program  management  was  able  to 
concentrate  on  managing  the  program  instead  of  spending  time 
justifying  its  continuation. 

The  first  programs  that  indicated  problems  with  concur¬ 
rent  acquisition  were  the  early  cruise  missile  programs. 

The  management  philosophies  incorporated  in  the  cruise 
missile  programs  were  significantly  different  from  the  ICBM 
program.  Concurrency  was  not  initially  planned,  but  was 
incorporated  due  to  unplanned  schedule  compression.  Also 
program  managers'  autonomy,  responsibility,  and  authority 
was  limited;  therefore,  reducing  their  ability  to  quickly 
deal  with  problems.  As  competition  for  limited  national 
resources  increased,  DOD  moved  to  an  acquisition  policy 
called  Total  Package  Procurement. 

Total  Package  Procurement  CTPP)  required  a  government 
commitment  to  production  at  the  time  of  contract  award  and 
was  concurrent.  TPP  limited  the  use  of  competition  during 
acquisition  and  increased  the  centralization  of  authority 
and  management  layers  in  the  acquisition  process.  Many 
systems  developed  under  TPP  experienced  cost  and  performance 
problems.  The  MBT-70  and  F-111B  programs  were  cancelled 
while  the  C-5A  and  F-lll  were  continued  to  production. 

In  the  1970s  Deputy  Secretary  of  Defense  David  Packard 
rejected  the  use  of  concurrency  and  required  the  use  of 
system  prototypes  and  a  sequential  acquisition  process.  The 
use  of  system  prototypes  has  led  to  the  production  of  some 
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very  good  systems  such  as  the  F-16,  A- 10,  and  the  avionics 
and  cannon  subsystems  of  the  F-15.  Despite  some  improve¬ 
ments  in  the  areas  of  program  management  and  system  testing, 
a  lengthening  acquisition  cycle  and  increasing  program  costs 
resulted  in  continued  evaluation  of  the  acquisition  cycle. 

Our  present  day  acquisition  policy  accepts  the  use  of 
concurrency  coupled  with  system  prototyping  when  appropri¬ 
ate.  The  amount  of  concurrency  and/or  the  decision  to  use 
prototypes  depends  on  the  level  of  technical  risk  and  the 
national  urgency  of  system  need.  Examples  of  programs  using 
concurrency  and  prototyping  are  the  B-1B,  F-16  C/D,  LANTIRN, 
and  AMR A AM. 

Benefits  of  Using  Concurrency.  The  primary  benefit  of 
a  concurrent  acquisition  program  was  demonstrated  by  the 
ICBM  development.  The  ICBM  program  was  expected  to  take 
from  10-11  years  to  complete;  however,  deployment  and  opera¬ 
tional  status  was  achieved  within  five  years-  The  1978 
Defense  Science  Board  reported  benefits  in  savings  in  total 
acquisition  costs  and  a  longer  useful  system  operational 
life  before  obsolesence.  Concurrency  can  also  provide  early 
system  design  maturity  and  identification  of  potential 
production  problems.  The  use  of  concurrency  does  not 
guarantee  success  and  program  managers  must  guard  against 
potential  problems. 

Problems  Associated  With  Concurrency.  Concurrency 
increases  program  risks.  In  the  overall  evaluation  concur- 
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rency  has  been  blamed  for  causing  the  same  problems  it  is 
credited  with  eliminating.  Due  to  concurrency's  shortened 
acquisition  cycle,  some  programs  have  produced  schedule 
slippages  and  increased  costs.  Therefore,  the  concurrent 
programs  took  too  long  to  develop  and  had  cost  overruns. 

GAO  reports  have  identified  additional  problems  as  inade¬ 
quate  test  data  for  production  decisions,  poor  system 
performance,  and  systems  deployed  with  known  problems 
requiring  system  design  and  retrofit.  Also,  concurrency  has 
been  identified  as  a  cause  of  poor/ low  system  reliability. 

Testing  R&M  Indicators  Versus  Field  R&M.  R&M  issues 
have  received  increasing  management  attention  over  the  past 
few  years.  As  the  cost  and  sophistication  of  weapon  systems 
have  increased,  the  need  to  improve  system  R&M  has  grown  in 
importance.  Today  acquisition  directives  require  that  R&M 
be  considered  equivalent  to  system  performance.  Therefore, 
system  R&M  must  be  tested  and  evaluated  during  the  acquisi¬ 
tion  process.  The  available  literature  indicates  that 
problems  exist  in  reaching  weapon  system  R&M  goals.  A  1977 
Hughes  Aircraft  Company  3tudy  of  16  different  avionics 
systems  showed  variances  between  predicted  and  field  relia¬ 
bility.  A  RAND  study  of  the  A-7D  also  demonstrated  signif¬ 
icant  differences  between  predicted  and  field  reliability. 

A  review  of  the  F-15  program  showed  early  operational 
problems  with  R&M.  Several  probable  causes  for  the 
disparity  between  R&M  measures  are  provided  in  the 
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literature.  The  reasons  for  low  and  inconsistent  system  R&M 
can  be  found  throughout  the  management  levels  of  the  acqui¬ 
sition  process. 

During  the  acquisition  process  numerous  program  deci¬ 
sions  are  made  which  can  impact  R&M.  These  decisions  are 
usually  driven  by  funding  and  schedule  constraints  which  may 
cause  a  negative  impact  on  R&M.  System  R&M  improvements  can 
appear  very  costly  when  viewed  in  the  short  term  development 
cycle.  Current  experience  shows  a  lack  of  congressional 
support  to  provide  the  upfront  funding  required  for  R&M 
improvements.  Also,  congressional  and  senior  management 
attention  has  placed  system  performance  as  the  primary 
element  in  determining  program  continuation  and  funding. 

This  mindset  has  placed  R&M  needs  behind  performance  needs 
during  development.  Another  issue  which  affects  R&M  and 
must  be  dealt  with  by  senior  Air  Force  leadership  is  in  the 
calculation  of  R&M  measures.  Although  guidelines  are 
provided  for  these  calculations,  the  definitions  used  to 
determine  what  constitutes  a  failure  are  different  between 
DT&E,  IOT&E,  and  the  field.  Therefore,  there  is  littie 
correlation  between  the  R&M  measures.  Attempts  to  reduce 
the  length  of  the  acquisition  cycle  create  additional 
problems  which  can  cause  a  reduction  in  field  R&M  measures 
when  compared  to  the  predicted  values.  Consequently, 
concurrency  has  been  considered  a  contributing  factor  to 
poor  R&M.  A  compressed  acquisition  schedule  may  result  in 


insufficient  system  testing  which  does  not  identify  system 
problems  in  time  to  fix  them  and/or  does  not  allow  time  to 
retest  fixes  to  prove  problems  have  been  corrected  before 
production.  Other  elements  which  affect  R&M  are  the  use  of 
contractor  personnel  and  support/test  equipment  during  DT&E 
and  IOT&E.  The  Hughes  study  showed  that  environmental 
differences  and  the  type  of  aircraft  in  which  a  system  is 
installed  can  affect  system  R&M.  During  system  acquisition 
it  is  important  for  managers  to  consider  all  these  areas 
with  the  potential  to  negatively  impact  R&M.  The  next 
section  provides  managers'  responses  outlining  concurrency's 
impact  on  their  program. 

Interview  Findings 

Program  Specif ics.  Interview  questions  2-6  were 
developed  to  provide  data  on  the  programs  used  in  this 
thesis.  This  data  was  needed  to  ensure  concurrency 's  use 
and  the  current  acquisition  phase  of  each  program. 

Four  programs  included  concurrency  in  the  initial 
planning.  The  remaining  program  incorporated  concurrency 
because  of  a  firm  IOC  date  and  schedule  slips  due  to  tech¬ 
nical  problems  and  budget  cuts.  According  to  the  inter¬ 
viewees,  only  one  program  was  considered  high  risk  at  the 
start  of  acquisition.  The  managers  interviewed  reported 
that  their  programs  overlapped  during  the  full  scale  devel¬ 
opment  and  production  phases.  All  programs  used  or  are 
scheduled  to  use  a  combined  DT&E  and  IOT&E  testing  procedure 
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which  in  three  out  of  five  cases  led  initially  to  low  rate 
production.  Full  rate  production  was  later  authorized  after 
additional  testing  and  development.  All  programs  are 
presently  engaged  in  some  development  and  production. 

Concurrency 's  Af f ect  on  Acquisition.  Interview  ques¬ 
tions  1,  16,  and  17  were  used  to  provide  information  on  the 

acquisition  managers'  general  opinions  of  concurrency.  Many 
of  the  managers  interviewed  stated  that  in  today  's  acquisi¬ 
tion  environment  concurrency  is  a  necessity.  One  manager 
replied  "It's  almost  mandatory  that  you  have  some  level  of 
concurrency  with  zero  concurrency  you  would  never  get  any¬ 
thing  done."  The  literature  recommends  that  to  be  success¬ 
ful,  concurrency  should  be  planned  from  the  beginning  of  a 
program.  Therefore,  the  decision  whether  or  not  to  use 
concurrency  should  be  made  as  early  in  the  acquisition  cycle 
as  possible.  This  decision  should  be  based  on  certain 
program  characteristics.  The  program  characteristics 
presented  by  the  managers  which  support  the  goals  of  a 
concurrent  program  are: 

1.  A  modification  of  an  old  system  or  a  new  system 
that  is  not  a  new  state  of  the  art  technology. 

2.  System  uses  rapidly  changing  technology  and  must 
be  deployed  quickly  to  prevent  obsolescence. 

3-  Program  is  low  risk  and/or  uses  proven 
technology. 

4.  Program  urgency  of  need  is  high  enough  to 
Justify  the  increased  risk  of  concurrency. 

5.  High  risk/new  technology  programs  should  not  use 
concurrency. 
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To  many  of  the  managers  interviewed,  concurrency  is 
neither  good  or  bad.  Its  value  is  dependent  on  the  system 
and  the  outcome  of  the  program.  For  some  the  success  of  a 
concurrent  program,  "...  depends  on  the  quality  of  manage¬ 
ment  in  the  SPO"  and  ”.  . .  the  proper  cooperation  between  the 
developers  and  testers".  The  bottom  line  on  concurrency  was 
best  summarized  by  one  manager: 

I  think  it  is  necessary.  It  has  some  disadvantages 
but  I  also  think  it  has  some  advantages.  It 's 
about  the  only  way  in  the  real  world  that  you  can 
do  it  [acquisition!.  You  don’t  have  any  choice  you 
use  it  to  stay  in  some  sort  of  budget  and  schedule. 

You  might  start  out  with  no  concurrency  but  before 
you  know  it  you  will  have  concurrency. 

Concurrency  offers  some  benefits  to  acquisition  programs. 

Benef its-  The  primary  benefit  of  concurrency 

identified  by  86  percent  of  the  interviewees  was  a  shorter 

acquisition  schedule.  Several  managers  provided  secondary 

benefits  that  are  derived  from  quicker  system  deployment. 

These  benefits  are: 

1.  Lower  acquisition  costs. 

2.  Reduces  the  chance  of  program  cancellation. 

3.  Provides  a  longer  system  useful  life. 

4.  Makes  it  easier  to  get  funding  and  avoid 
political  turmoil. 

One  manager  felt  that  concurrency  could  minimize  the  affects 

of  some  problems  on  the  overall  program. 

Concurrency  allows  you  to  meet  a  schedule.  If  you 
have  problems  in  development  and  I'm  not  saying 
development  problems,  but  in  our  case  funding 
problems  that  cause  you  to  stretch  out  development, 

...  you  can  still  meet  the  end  item  schedule,  IOC. 
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Two  managers  considered  earlier  involvement  of  user  and 
operational  personnel  as  an  important  aspect  of  concurrent 
programs,  "It  is  important  in  that  it  provides  the  ability 
to  identify  operational  problems  early".  Besides  benefits 
concurrency 's  use  results  in  increased  program  risk.. 

Problems.  The  potential  for  problems  in  concur¬ 
rent  programs  was  expressed  by  one  of  the  interviewees, 
"Concurrency  provides  a  shorter  schedule  . . .  You  can  always 
go  back  and  fix  it  once  you  have  it".  While  a  shorter 
acquisition  cycle  is  considered  a  benefit,  it  is  also  a 
source  of  several  potential  problems  during  concurrency. 
Less  time  to  develop  a  system  may  translate  into  a  shorter 
testing  phase  which  creates  problems  as  DT&E  and  IOT&E  may 
or  may  not  be  completed  prior  to  the  production  start.  Two 
major  logistics  areas  are  impacted  by  this  timing  problem. 
As  one  manager  stated: 

We're  constantly  slightly  late  getting  the  right 
configuration  of  support  and  test  equipment  out  to 
meet  the  system  and  that  is  hampered  by  concurrency 
because  we're  just  barely  finishing  up  development 
before  we  start  production. 

Some  additional  areas  of  concern  pointed  out  by  the  manager 
were: 

1.  Final  item  design  may  not  be  completed  before 
production. 

2.  Systems  may  be  delivered  with  some  equipment 
not  available. 

3.  Limits  ability  to  complete  R&M  testing  and 
incorporate  fixes  before  production. 
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4.  Fixes  designed  to  correct  problems  found  in 
DT&E  and  IOT&E  may  not  be  retested  before 
production. 

A  further  review  of  the  interview  responses  indicated  the 

need  for  system  retrofits  as  the  most  reported  problem. 

According  to  one  manager,  "Mostly,  you'll  find  things  that 

just  don't  work.  It  will  be  mandatory  to  go  back  and 

retrofit  sometimes".  Another  manager  provided  this  comment: 

You  buy  some  problems  with  concurrency.  Things 
will  get  out  of  sync  and  you'll  be  producing 
hardware  and  not  be  able  to  get  all  the  changes 
into  it  C system! . 

The  problem  of  system  retrofit  was  a  major  concern  of  the 
SPO  personnel  interviewed.  This  concern  with  system  retro¬ 
fit  is  understandable  since  the  SPO  has  overall  system 
responsibi 1 ity. 

Concurrency  creates  problems  for  program  management 
with  many  simultaneous  activities  requiring  management 
attention.  These  include  development  and  reliability 
testing,  flight  and  ground  testing  for  DT&E  and  IOT&E,  R&M 
testing,  and  system  integration.  Many  of  these  tasks  impact 
the  accomplishment  of  AFOTEC 's  mission.  Consequently, 

AFOTEC  personnel  identified  management  of  concurrent  issues 
as  a  problem  when  an  acquisition  program  has  limited  assets. 

If  acquisition  managers  fail  to  eliminate  or  minimize 
the  potential  problems  of  concurrency,  the  affects  on  a 
weapon  system  can  be  serious.  An  acquisition  manager 
explained: 
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...  an  unstable  configuration  ...  a  lot  of  changes 
.  .  .  some  of  those  can  be  costly  .  .  .  you  have  more 
ECPs  and  the  program  is  a  little  more  turbulent. 

. . .  you  could  end  up  with  a  design  that  could  not 
do  the  job  either  from  a  performance  or  R&M  stand¬ 
point. 

The  primary  goal  of  concurrency,  a  shorter  schedule,  is 
reportedly  incompatible  with  the  goal  of  improving  system 
R&M.  Therefore,  the  researcher  developed  several  interview 
questions  to  evaluate  concurrency 's  impact  on  R&M. 

Concurrency 's  Impact  On  System  R&M.  Since  most  of  the 
available  literature  did  not  provide  positive  R&M  benefits 
of  concurrent  programs,  the  managers  were  asked  to  comment 
on  this  area  through  interview  question  15.  To  obtain 
information  on  the  reported  factors  which  may  negatively 
impact  system  R&M,  interview  questions  7-14  and  hypotheses 
1-8  were  developed. 

When  asked  to  provide  information  on  how  concurrency 

benefits  R&M,  only  four  managers  gave  a  positive  response. 

Three  of  them  identified  the  early  involvement  of  user  and 

operational  personnel  to  evaluate  the  system  as  a  benefit. 

Two  other  managers  although  they  did  not  feel  concurrency 

benefited  R&M  did  provide  some  possible  benefits. 

Maybe  through  schedule  savings  or  savings  in  money 
so  you  can  do  other  things  you  might  want  to  do. 

...  one  possible  way,  that  while  the  guys  are 
producing  it  the  people  in  development,  have  a 
closed  feedback  between  the  two  and  they  can  make 
corrections. 

While  10  of  the  15  managers  interviewed  stated  that  concur¬ 
rency  did  not  benefit  R&M,  one  manager's  response  seemed  to 
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summarize  the  groups'  thoughts  on  this  issue-  He  stated, 

"In  concurrency  you  do  things  to  benefit  cost  and  schedule, 
the  problems  that  arise  are  usually  R&M". 

Over  the  last  several  years  one  of  the  major  issues  of 
DOD  acquisition  programs  has  been  to  improve  weapon  system 
R&M.  Numerous  factors  have  been  identified  as  causing  the 
poor  correlation  between  R&M  measures  reported  for  DT&E/ 
IOT&E  and  the  actual  field  demonstrated  R&M  of  many  weapon 
systems  and  subsystems.  Several  of  these  factors  were 
reviewed  in  Chapter  II  and  will  be  investigated  further  in 
the  remainder  of  this  chapter. 

Contractor  Personnel.  All  the  acquisition 

programs  used  in  this  thesis  utilized  contractor  personnel 

to  maintain  their  systems  during  the  FSD  phase-  However, 

four  of  the  programs  also  used  Air  Force  maintenance 

personnel.  When  asked  if  the  use  of  contractor  maintenance 

during  DT&E/ IOT&E  caused  a  reduction  in  field  R&M  measures, 

nine  managers  (60  percent)  said  it  did  Csee  Table  2). 

The  responses  to  Hypothesis  1  appear  to  support  the 

literature  which  states  that  the  use  of  contractor  personnel 

may  negatively  impact  R&M.  However,  the  managers  felt  that 

any  bias  of  R&M  data  was  minimal.  The  dominant  reason  given 

for  the  R&M  bias  was  the  3kill  and  experience  differences 

between  the  contractor  personnel  and  the  user  command 

maintenance  technicians.  As  one  manager  put  it: 

The  contractor  has  to  bias  it  a  little  because  he 
has  highly  qualified  people  compared  to  the  average 
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GI.  He  most  likely  has  higher  paid  more  experienced 
people  and  that  would  make  the  reliability  and 
maintainability  picture  a  little  brighter. 


Table  2 

Hypothesis  1  Interview  Question  7 


Hypothesis:  Use  of  contractor  maintenance  personnel  for 

system  maintenance  reduces  the  accuracy  of  system  R&M 
measures. 


Was  contractor  maintenance  used 
during  DT&E  and  IOT&E? 

YES  NO 

Reduces  accuracy  of  YES  9 

system  R&M  measures 

NO  6 


According  to  another  manager  contractor  maintenance  prima¬ 
rily  affected  system  maintainability. 

Your  going  to  have  a  failure  regardless  of  who 
fixes  it,  but  then  you  might  not  necessarily  see 
some  of  the  problems  with  your  support  equipment  or 
...  training.  Your  equipment  will  get  fixed 
quicker  than  in  the  real  world. 

The  impact  to  system  R&M  of  this  factor  will  be  eliminated 
as  Air  Force  personnel  acquire  experience  explained  one  SPO. 
An  analysis  of  the  six  managers  who  reported  no  bias  showed 
that  two  were  involved  with  a  program  which  exceeded  DT&E 
and  IOT&E  R&M  in  the  field.  Also,  four  of  the  five  DPML 
personnel  interviewed  did  not  consider  contractor  mainten¬ 
ance  a  problem.  All  five  AFOTEC  personnel  reported  that  the 
use  of  contractor  personnel  will  impact  R&M. 
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Support  Equipment ■  Many  acquisition  programs  do 
not  have  fully  developed  support  equipment  available  during 
FSD  and  this  may  impact  system  R&M.  Hypothesis  2  was 
designed  to  determine  the  support  equipment  status  on  the 
programs.  Review  of  the  responses  showed  that  in  four 
programs  all  three  managers  agreed  on  the  question  of 
equipment  availability  during  DT&E  and  IOT&E.  However,  only 
one  program  had  all  managers  report  the  same  impact  to  R&M. 
On  this  program  all  managers  felt  that  the  nonavailability 
of  support  equipment  would  reduce  field  R&M.  One  manager 
did  not  respond  to  this  question  Csee  Table  3). 

Table  3 

Hypothesis  2  Interview  Question  8 

Hypothesis:  Lack  of  system  unique  support  equipment  during 

testing  reduces  the  accuracy  of  system  R&M  measures. 

Was  system  unique  support  equipment 
available  during  DT&E  and  IOT&E? 

YES  NO 

Reduces  accuracy  of  YES  1  6 

system  R&M  measures 

NO  3  4 

Three  programs  were  identified  as  not  having  the  field 
projected  support  equipment;  however,  the  managers  disagreed 
on  the  impact  to  R&M.  With  10  managers  reporting  a  lack  of 
equipment  60  percent  felt  it  would  reduce  field  R&M.  The 
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responses  to  this  hypothesis  were  balanced  between  all 
management  areas.  The  managers  stated  that  either 
contractor  or  some  type  of  modified  equipment  was  used. 

Test  Equipment .  As  weapon  systems  become  more 
integrated,  the  dependence  on  software  increases.  This 
trend  results  in  the  simultaneous  development  of  both 
software  and  hardware  for  the  weapon  system  and  the  test 
equipment.  The  reduced  acquisition  time  of  a  concurrent 
program  will  increase  the  probability  that  test  equipment 
will  not  be  fully  developed  for  use  in  DT&E/IOT&E.  The  lack 
of  field  test  equipment  has  been  identified  as  having  a 
negative  impact  on  R&M  correlation  between  the  testing  and 
operational  environments.  Hypothesis  3  was  designed  to 
investigate  this  issue  Csee  Table  4).  One  manager  did  not 
respond  to  the  question. 


Table  4 

Hypothesis  3  Interview  Question  9 


Hypothesis:  Lack  of  system  unique  test  equipment  during 

testing  reduces  the  accuracy  of  system  R&M  measures. 


Was  system  unique  test  equipment 
available  during  DT&E/IOT&E? 

YES  NO 

Reduces  accuracy  of  YES  1  6 

system  R&M  measures 

NO  3  4 
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The  managers  in  three  programs  did  not  agree  on  the  availa¬ 
bility  of  test  equipment  during  their  program's  DT&E  and 
IOT&E.  There  were  no  programs  where  al 1  three  managers 
agreed  on  the  impact  of  test  equipment  on  R&M  measures.  Ten 
managers  acknowledged  that  the  field  system  unique  test 
equipment  was  not  available  during  system  testing.  While  60 
percent  reported  this  would  have  a  negative  impact.  In  this 
group  three  of  four  DPMLs  did  not  feel  it  would  negatively 
impact  R&M  while  the  three  AFOTEC  personnel  felt  the  lack  of 
test  equipment  would  result  in  lower  R&M.  Seven  managers' 
response  to  Hypothesis  2  matched  their  response  to  this 
issue. 

Tech  Data.  The  lack  of  technical  data  during 
weapon  system  testing  may  impact  system  R&M.  Technical  data 
development,  like  test  equipment,  is  affected  by  concurrency 
and  the  level  of  software  in  a  weapon  system.  Hypothesis  4 
deals  with  the  availability  of  tech  data  during  DT&E/IOT&E. 
Two  managers  did  not  answer  this  question.  In  four  programs 
the  managers  responding  agreed  on  the  availability  of  tech 
data  during  testing.  However,  only  two  sets  of  managers 
agreed  on  the  Impact  to  their  system.  In  one  case  tech  data 
was  available  with  no  impact  to  R&M.  While  the  other 
program  reported  no  tech  data  and  the  managers  felt  this 
would  reduce  R&M  measures.  Eight  managers  reported  a  lack 
of  tech  data  during  their  program  testing  but  only  four, 
three  from  AFOTEC,  considered  it  a  problem  for  R&M.  Also, 


in  two  cases  both  the  SPO  and  DPML  reported  no  impact  while 
their  AFOTEC  counterpart  felt  R&M  would  be  reduced  in  the 
field.  In  the  opinion  of  one  SPO  the  lack  of  tech  data 
during  testing  coupled  with  its  availability  later  in  the 
field  would  have  a  positive  affect  on  field  R&M  measures 
Csee  Table  5). 


Table  5 

Hypothesis  4  Interview  Question  10 

Hypothesis:  The  lack  of  system  technical  data  during  DT&E 
and  IOT&E  reduces  the  accuracy  of  system  R&M  measures. 

Was  tech  data  available  during 
DT&E/ IOT&E? 

YES  NO 

Reduces  accuracy  of  YES  4 

system  R&M  measures 

NO  5  4 

Incomplete  DT&E/ IOT&E.  Several  studies  have  indi¬ 
cated  that  concurrent  programs  produce  incomplete  testing, 
may  delay  DT&E  to  complete  IOT&E,  and/or  fail  to  identify 
system  problems  before  production.  Data  was  collected  to 
determine  whether  or  not  the  five  concurrent  programs  used 
in  this  study  experienced  any  of  these  problems.  All 
programs  used  a  combined  DT&E/ IOT&E  phase  that  is  that  some 
DT&E  and  IOT&E  testing  was  being  accomplished  at  the  same 
time.  As  a  group  the  managers  did  not  feel  that  DT&E  was 
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delayed  to  complete  IOT&E.  According  to  one  manager  in  his 
program  IOT&E  had  been  delayed  for  DT&E.  Other  managers 
reported  finding  problems  which  caused  a  delay  in  the  full 
rate  production  decision.  Hypothesis  5  was  designed  to  see 
if  production  decisions  were  made  with  incomplete  DT&E/ 
IOT&E.  In  response  to  this  hypothesis  the  three  managers  in 
four  programs  all  agreed  to  the  status  of  DT&E/IOT&E  at  the 
time  of  their  program's  production  decision.  In  two  cases 
the  managers  agreed  on  the  impact  to  R&M.  Analysis  of  the 
responses  to  this  hypothesis  shows  eight  managers  in  pro¬ 
grams  where  a  production  decision  was  made  before  completion 
of  DT&E/IOT&E  with  88  percent  reporting  a  reduction  in  field 
R&M  measures.  The  remaining  seven  managers  felt  DT&E/IOT&E 
were  completed  before  the  production  decision  and  85  percent 
reported  that  this  would  not  reduce  field  R&M  (see  Table  6). 

Table  6 

Hypothesis  5  Interview  Question  11 

Hypothesis:  Incomplete  DT&E  and  IOT&E  testing  prior  to  the 

production  decision  reduces  the  accuracy  of  system  R&M 
measures. 


Was  production  decision  made  with 
incomplete  DT&E/IOT&E? 

YES  NO 

Reduces  accuracy  of  YES  7  1 

system  R&M  measures 

NO  1  6 
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Design  Freeze.  According  to  the  literature,  an 


early  design  freeze  will  negatively  impact  the  relationship 

between  testing  and  field  R&M.  Interview  question  12  and 

Hypothesis  6  were  developed  to  look  at  this  area.  In 

response  to  the  issue  of  design  freezes  67  percent  of  the 

managers  did  not  think  that  concurrency  caused  an  early 

freeze.  Several  managers  reported  that  their  programs  have 

continued  to  have  design  changes  with  some  occurring  up  to 

and  surpassing  the  start  of  production.  One  manager  said: 

The  design  has  undergone  several  refinements,  some 
performance  related  as  a  result  of  original  testing 
phases.  Other  significant  changes  were  done  for 
reasons  of  cost  and  manufacturing  effectiveness. 

Expressing  a  different  idea  one  manager  explained  that  con¬ 
currency  prevented  a  design  freeze  until  after  production. 
This  resulted  in  retrofits  to  some  production  items.  The 
remaining  five  managers  considered  concurrency  as  a  cause  of 
early  design  freezes  and  their  thoughts  are  reflected  in 
this  manager's  comments: 

...  after  the  design  freeze  your  still  finding 
problems  which  if  they  prove  significant  may  result 
in  retrofits.  ...  some  unquant i f iable  long  term 
effect  on  R&M  because  you  could  not  get  some 
changes  in  that  you  would  like. 

Only  13  managers  were  sure  about  their  program's  use  of  a 

design  freeze.  The  results  of  the  responses  showed  four 

programs  where  the  managers  agreed  on  whether  or  not  the 

system  experienced  a  design  freeze.  However,  the  managers 

of  only  two  programs  concurred  on  the  impact  of  this  action. 

Table  7  provides  the  data  collected  for  Hypothesis  6. 
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Table  7 


Hypothesis  6  Interview  Question  11 


Hypothesis:  Early  system  design  freeze  for  IOT&E  will 

reduce  the  accuracy  of  system  R&M  measures. 


Was  there  an  early  design  freeze  to 
complete  IOT&E  testing  for  the 
production  decision? 

YES  NO 

Reduces  accuracy  of  YES  1  2 

system  R&M  measures 

NO  5  5 


Overall  77  percent  of  the  responding  interviewees  did  not 
feel  that  having  or  not  having  a  design  freeze  for  IOT&E 
would  result  in  any  problem  for  field  R&M  measures.  This 
result  supports  the  data  collected  for  interview  question  12 
and  disagrees  with  the  available  literature. 

Engineering  Change  Proposals.  Interview  question 
13  and  Hypothesis  7  looked  at  the  potential  impact  of  open/ 
untested  ECPs  on  system  R&M.  Some  managers  were  unsure  of 
the  status  of  ECPs  at  the  time  of  IOT&E  and  the  production 
decision  and  did  not  answer.  The  other  managers  indicated 
that  it  is  normal  practice  to  have  open/ untested  ECPs  going 
into  both  IOT&E  and  the  production  decision.  The  ECPs  were 
not  only  for  reliability  issues  but  also  dealt  with  perform- 
mance,  producibi 1 ity ,  and  maintainability.  A  few  managers 
pointed  out  that  their  program  had  some  ECPs  dealing  with 
support/te3t  equipment.  The  ECPs  were  usually  a  result  of 
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problems  found  in  DT&E  that  were  not  corrected  in  time  for 
IOT&E.  On  one  program  many  of  the  ECPs  were  not  going  to  be 
fully  implemented  until  production  Csee  Table  8). 


Table  8 

Hypothesis  7  Interview  Question  13 


Hypothesis:  Open/untested  ECPs  for  reliability  improvements 

at  the  time  of  production  decision  reduces  the  accuracy  of 
system  R&M  measures. 


Did  open/ untested  ECP  reliability 
improvements  exist  at  the  time  of 
the  production  decision? 

YES  NO 

Reduces  accuracy  of  YES  5 

system  R&M  measures 

NO  4  2 


Of  the  11  managers  who  responded  to  the  hypothesis,  82 
percent  reported  a  production  decision  with  open/untested 
ECPs.  However,  only  56  percent  felt  this  would  result  in 
reduced  R&M  measures  in  the  field.  When  looking  at  the  data 
to  see  if  the  individual  program  managers  agreed,  only  two 
programs  existed  with  all  managers  agreeing  on  the  status  of 
ECPs.  There  were  no  programs  where  all  managers  reported 
the  same  impact.  Further  review  showed  that  an  individual 's 
response  was  influenced  by  his  area  of  responsibility.  All 
three  AFOTEC  respondents  thought  open/untested  ECPs  reduced 
field  R&M  measures. 
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Test.  Analyze,  and  Fix.  The  literature  proposes 
the  use  of  TAAF  as  a  very  important  means  of  developing 
reliability  improvements  and  indicates  that  concurrency  may 
reduce  its  use.  Interview  question  14  and  Hypothesis  8  are 
presented  in  this  thesis  to  investigate  this  issue.  During 
the  interviews  one  manager  did  not  feel  his  program  used  a 
TAAF  phase  and  two  others  were  not  sure  how  complete  their 
program's  testing  was  during  development.  Therefore,  these 
managers  did  not  answer  the  questions.  Analyzing  the 
responses  to  the  question  of  whether  or  not  concurrency 
impacted  their  TAAF  phase  showed  that  64  percent  of  the 
managers  did  not  see  concurrency  as  a  problem.  The  comments 
of  one  manager  seemed  to  explain  the  use  of  a  TAAF  phase  in 
a  concurrent  program: 

...  what  your  going  to  find  is  we're  out  there 
flying  equipment  while  we're  still  in  the  lab  doing 
reliability  growth  programs.  So  we're  3till  using 
it  but  it's  Just  a  concurrent  phase-  We  have  lab 
testing  going  on  at  the  same  time  we  have  opera¬ 
tional  testing  going  on. 

This  statement  al3o  provides  a  look  at  the  general  response 
for  the  managers  who  reported  that  concurrency  does  impact  a 
TAAF  program.  The  impact  was  experienced  in  the  amount  of 
time  to  perform  the  TAAF  phase.  As  one  manager  stated, 
"There  were  some  technical  problems  experienced  and  this 
resulted  in  . . .  maturity  problems.  Some  fixes  had  to  be 
checked  in  IOT&E".  Several  managers  expressed  the  opinion 
that  a  TAAF  phase  was  a  very  cost  effective  means  of  finding 
system  problems.  There  were  other  issues  presented  by  the 
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managers  that  can  impact  a  TAAF  program.  These  include: 
limited  test  assets,  a  reluctance  to  fix  identified  problems 
due  to  cost,  and  the  potential  of  a  program  being  cancelled 
because  of  system  failures  during  the  TAAF  phase.  Of  the 
five  programs  used  in  this  thesis  three  sets  of  acquisition 
managers  agreed  on  the  status  of  TAAF  during  development  and 
TAAF 's  impact  on  their  program.  In  all  three  cases  the 
programs  had  an  incomplete  TAAF  phase  during  development  and 
the  managers  felt  this  would  reduce  the  R&M  measures  in  the 
field.  This  response  supports  the  researcher's  findings  in 
the  literature.  Looking  at  the  managers'  answers  to  the 
hypothesis  shows  that  67  percent  reported  an  incomplete  TAAF 
phase  in  development  with  88  percent  indicating  this  will 
have  a  negative  impact  on  R&M  Csee  Table  9). 

Table  9 

Hypothesis  8  Interview  Question  14 

Hypothesis:  An  Incomplete  test,  analyze,  and  fix  program 
during  development  reduces  the  accuracy  of  system  R&M 
measures. 


Vfas  there  an  incomplete  test, 
analyze,  and  fix  during 
development? 

YES  NO 

Reduces  accuracy  of  YES  7  1 

system  R&M  measures 

NO  1  3 
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Program  Results.  One  of  the  main  intents  of  this 
research  was  to  determine  the  extent  that  the  factors 
identified  in  the  literature  influenced  concurrent  programs 
in  the  area  of  R&M.  A  secondary  goal  was  to  see  if  acquisi¬ 
tion  managers  in  the  same  program  but  from  different  major 
areas  of  concern  for  system  development  had  similar  opinions 
of  their  program. 

An  analysis  of  the  acquisition  managers'  responses  was 
conducted  to  determine  the  applicability  and  impact  on  R&M 
of  the  hypothesis  factors  from  a  program  standpoint.  The 
factors  were: 

1.  Contractor  maintenance  used  during  DT&E/IOT&E. 

2-  Lack  of  system  unique  support  equipment  during 
testing. 

3.  Lack  of  system  unique  test  equipment  during 
testing. 

4.  Lack  of  system  technical  data  during  DT&E/IOT&E. 

5.  Incomplete  DT&E/IOT&E  prior  to  production  decision. 

6.  Early  system  design  freeze  for  IQT&E. 

7.  Open/untested  ECPs  at  the  time  of  production 
decision. 

8.  Incomplete  TAAF  phase  during  development. 

In  order  to  accomplish  this  analysis,  the  researcher  had  to 
make  some  assumptions  about  the  recorded  data.  Most  inter¬ 
viewees  had  at  least  one  to  three  years  on  their  program 
with  only  two  having  less  than  six  months.  Some  acquisition 
managers  could  not  answer  all  the  hypothesis  questions  in 
terms  of  their  program.  The  fact  that  current  acquisition 
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programs  take  from  10-12  years  and  personnel  turnover 
accounts  for  some  of  this  data  loss.  Another  issue 
requiring  adjustment  was  identified  when  looking  at  manager 
concurrence  on  the  factor 's  status  and  impact  on  the 
individual  programs.  With  eight  hypotheses  presented  in 
five  programs,  a  total  of  40  answers  could  have  reflected 
the  concurrence  of  three  managers.  However,  the  answers  to 
the  hypotheses  showed  that  three  managers  agreed  only  four 
times  when  the  responses  for  all  five  programs  were  totaled. 
Further  analysis  showed  that  a  total  of  30  times  at  least 
two  managers  provided  the  same  response.  While  the 
disparity  between  the  potential  responses  and  the  actual 
responses  can  be  somewhat  explained  by  the  13  times  managers 
did  not  reply  to  the  hypotheses,  it  does  not  explain  it  all. 
In  order  to  evaluate  the  factors  in  terms  of  each  program 
the  researcher  used  the  hypothesis  response  provided  by  at 
least  two  of  the  three  managers  to  represent  the  program. 

If  at  least  two  common  responses  could  not  be  found  in  terms 
of  factor  status  and/or  impact  on  R&M,  the  program  was  not 
counted  in  the  results.  Therefore,  the  analysis  of  some 
factors  will  show  less  than  five  programs. 

Contractor  Personnel.  All  five  acquisition 
programs  used  contractor  maintenance.  In  four  programs  at 
least  two  of  the  three  managers  felt  it  would  reduce  the 
accuracy  of  R&M  measures  in  the  field  when  compared  to  test 
results.  The  other  managers  felt  there  was  no  R&M  impact. 
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Support  Equipment .  Four  programs  were 
evaluated  for  this  issue.  In  three  programs  system  unique 
support  equipment  was  not  available  during  testing.  The 
managers  in  two  programs  reported  that  this  would  impact  R&M 
and  one  program’s  managers  expected  no  impact.  In  one 
program  the  support  equipment  was  available. 

Test  Equipment .  Of  the  four  programs  used  to 
provide  data  on  this  factor,  three  confirmed  that  system 
unique  test  equipment  was  not  available  during  testing  and 
that  this  would  negatively  impact  system  R&M.  Test  equip¬ 
ment  was  available  in  the  fourth  program. 

Tech  Data-  Three  of  five  programs  did  not 
have  tech  data  during  testing-  Management  in  two  of  these 
programs  felt  the  lack  of  tech  data  would  not  impact  R&M. 
Only  one  program  reported  that  a  lack  of  tech  data  would 
reduce  R&M  measures'  accuracy.  Tech  data  was  available 
during  testing  in  one  program. 

DT&E/ IQT&E.  The  management  responses  showed 
that  in  three  programs  the  production  decision  was  made  with 
incomplete  DT&E/IGT&E  and  this  would  reduce  the  accuracy  of 
R&M  measures.  Two  programs  were  able  to  complete  DT&E/IOT&E 
requirements  before  the  production  decision. 

Design  Freeze.  The  data  showed  three 
programs  c.id  not  have  a  design  freeze  for  IOT&E.  Two 
programs  did  have  a  design  freeze.  In  two  programs  the 
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impact  of  this  factor  could  not  be  determined.  The  other 
program  managers  did  not  feel  this  factor  would  affect  R&M. 

ECPs.  Three  programs  did  have  open/untested 
ECPs  at  the  time  of  the  production  decision  with  one 
reporting  no  impact  to  R&M.  Due  to  the  disparity  of  the 
managers'  answers,  two  programs  were  not  counted  at  all  and 
the  impact  on  R&M  for  two  programs  could  not  be  determined. 

TAAF.  The  data  on  three  programs  showed  that 
there  was  an  incomplete  TAAF  phase  during  development.  The 
managers  felt  that  it  would  result  in  reduced  R&M  field 
measures.  The  remaining  two  programs  were  able  to  complete 
their  TAAF  programs. 

This  review  of  the  data  collected  in  response  to  the 
eight  hypothesis  questions  showed  that  all  but  one  of  the 
hypothesis  conditions  were  present  in  at  least  three  of  the 
five  programs.  The  factor  of  a  design  freeze  for  IOT&E  was 
only  used  in  two  programs.  Three  of  five  programs  supported 
the  thesis  hypotheses  dealing  with  the  factors  of  test 
equipment,  incomplete  DT&E/ IOT&E,  and  incomplete  TAAF.  The 
hypothesis  covering  the  use  of  contractor  maintenance  was 
confirmed  in  four  of  the  five  programs. 

Managers  '  Opinions  on  the  Dif  f erence  Between  DT&E/ 

IOT&E  and  Field  R&M  Measures.  The  hypotheses  developed  in 
thi3  thesis  did  not  cover  all  the  reasons  for  the  difference 
between  testing  and  field  R&M  results  identified  in  the 


literature.  Therefore,  interview  question  19  was  included 

to  obtain  additional  data  on  this  issue. 

Many  of  the  managers  identified  the  same  causes.  Six 

managers  identified  the  differences  in  maintenance  skill  and 

expertise  between  the  technicians  used  in  DT&E/IOT&E  and  the 

field  personnel.  As  one  manager  commented: 

First,  you  have  experienced  contractors  who  know 
the  system  inside  and  out  and  have  special  test 
equipment.  Second,  you  have  a  trained  group  of 
blue  suit  maintenance  personnel  who  are  better  than 
the  normal  user  command  personnel. 

Another  reason  given  for  the  poor  correlation  in  R&M  data, 

was  the  difference  in  the  testing  and  field  environment.  It 

is  not  possible  to  duplicate  the  weapon  system's  operational 

environment  during  development.  The  use  of  IOT&E  is  usually 

representative  but  still  lacks  some  key  factors.  A  total  of 

five  managers  reported  this  factor.  One  manager  stated: 

The  difference  is  in  the  labs  and  real  world 
environment.  There's  a  lot  of  procedures  that 
we 've  agreed  to  during  those  phases  tnat  perhaps 
misrepresent  the  numbers.  It's  basically  just  an 
environment  difference... 

The  maintenance  environment  of  the  system  is  also  a  factor 
to  consider.  Many  development  programs  have  dedicated 
facilities  which  allow  most  of  the  maintenance  to  be 
performed  inside.  This  way  the  developmental  system  is  not 
exposed  to  the  weather  as  frequently  as  the  operational 
system.  While  there  is  little  that  can  be  done  to  make  the 
development  environment  more  operationally  representative 
and  it  appears  the  use  of  contractor  maintenance  is  *he 
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norm.  The  impact  to  R&M  measures '  accuracy  created  by  data 
collection  can  be  worked. 

The  issue  of  data  collection  during  development  testing 

was  acknowledged  by  three  managers  as  a  source  of  the  R&M 

disparity.  One  manager  explained  it  as : 

The  data  collection  early  on  with  the  contractor 
and  SPO  guys  is  more  likely  to  be  flavored  to 
getting  the  system  to  look  as  good  as  possible. 

Let's  say  you  have  a  failure  and  you  don't  know  if 
it 's  type  1  or  not.  We  use  JRMET  . . .  and  sometimes 
give  the  system  the  benefit  of  the  doubt.  We  get 
rid  of  a  lot  of  faults  that  should  not  be  blamed  on 
the  system  your  testing. 

As  this  manager  further  noted,  the  field  units  do  not  use 

anything  like  JRMET.  They  also  do  not  deai  with  failures  in 

the  way  expressed  by  another  manager  who  reported; 

. . .  usually  in  our  lao  testing  we  tend  to  rule  out 
a  certain  amount  of  failures.  If  there  is  a 
failure  and  a  fix  is  identified  but  has  not  been 
incorporated  and  an  item  fails  a  second  time  for 
the  same  cause,  we  don't  count  it. 

Several  other  probable  causes  were  presented  during  the 

interviews-  These  were; 

1.  A  smaller  number  of  systems  to  maintain  in 
development . 

2.  Contractor  uses  different  test  and  support 
equipment . 

3.  Problems  with  initial  production  run  like  quality. 

4.  A  spares  shortfall  until  system  reaches  maturity. 

5.  Lack  of  formalized  follow-up  of  predicted  R&M  data. 
The  mos*  frequently  identified  cause  for  the  gap  in  testing 
and  field  R&M  data  results  from  the  numerous  definitions 
used  in  R&M  calculations  and  the  measures  themselves. 
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The  issue  of  definitions  and  understanding  the  R&M 
program  was  discussed  in  seven  of  the  interviews.  "There  is 
a  difference  in  what  we're  measuring.  The  definition 
changes  from  DT&E  to  operational.",  explained  one  manager. 
The  R&M  calculations  are  affected  by  the  difference  in  the 
time  factor.  In  DT&E  time  is  usually  total  system  operating 
time  while  in  the  field  flying  time  is  used.  Several 
managers  feel  a  standard  definition  is  needed.  Another 
manager  explained,  "the  user 's  measures  of  merit  are 
different  than  what  we  hold  the  contractor  responsible  for. 
There  is  no  direct  translation  between  what  the  user  and 
contract  consider  important.  " 

This  last  comment  raises  an  important  issue.  Are  the 
systems  being  developed  to  the  level  of  R&M  that  the  opera¬ 
tional  command  wants?  The  literature  reported  that  many 
systems  are  not  reaching  the  desired  R&M  levels.  However, 
many  systems  are  meeting  the  contract  requirements.  In  the 
opinion  of  three  managers  field  R&M  is  as  high  as  the 
development  predictions  but  this  fact  is  not  known  because 
of  the  way  field  data  is  handled.  Another  manager  was 
concerned  with  the  misunderstanding  of  reliability  growth 
and  reliability  maturity.  Many  people  do  not  understand 
that  the  predicted  reliability  given  for  a  system  is  based 
on  a  mature  system.  The  maturity  of  a  new  system  is  based 
*■  a  total  number  of  flying  hours  and  to  reach  this  goal  may 
•  i- *•  i  :*w  years  after  initial  system  delivery.  In  the  mean 
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time  the  system  and  subsystems  reliability  should  be 
compared  to  the  projected  reliability  growth  curve.  The 
mature  system  R&M  values  will  not  be  achieved  with  the  first 
systems  deployed.  The  importance  of  understanding  and 
standardizing  the  R&M  definitions  was  provided  by  an  acqui¬ 
sition  manager  when  he  stated,  "Half  the  problem  with  the 
R&M  program  is  understanding  the  definitions".  After 
discussing  the  problems  and  reasons  for  poor  R&M,  the  next 
area  to  look  at  is  how  to  improve  R&M. 

Improving  System  R&M.  Interview  question  18  was  used 
to  provide  the  data  for  this  section.  The  problem  of  low 
system  R&M  must  continue  to  receive  top  level  management 
attention.  System  R&M  is  to  be  considered  equal  to  cost, 
schedule,  and  performance  during  acquisition  programs. 

These  statements  provide  the  answer  to  improving  R&M 
according  to  three  of  the  acquisition  managers  interviewed. 
While  R&M  2000  and  other  directives  have  continued  to  push 
system  R&M,  the  interviews  indicated  that  more  emphasis  is 
needed.  The  response  from  one  manager  indicated  that 
improving  R&M  still  comes  down  to  the  issues  of  money  and 
priorities.  He  suggested: 

More  money  has  to  be  allocated  for  that  purpose 
CR&M1 .  A  lot  of  times  when  we  get  into  a  crunch 
for  what  ...  to  spend  our  bucks  for,  it  is  easier 
to  Justify  changes  for  performance  than  R&M. 

This  same  manager  provided  an  explanation  as  to  why 

performance  is  above  R&M  when  he  commented: 
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You  want  the  system  to  perform  first  then  you  can 
work  the  problems  of  how  to  fix  it  and  make  it  last 
longer.  You  can  delay  R&M.  .  .  .  The  whole  system  is 
geared  that  way.  Your  program  will  be  killed 
quicker  for  performance  problems  than  for  R&M. 

Placing  a  higher  priority  on  performance  than  R&M  early  in 

the  acquisition  program  is  the  type  of  decision  which  may 

prevent  significant  R&M  improvements.  Managers  have  to  put 

emphasis,  time,  and  money  towards  R&M  issues  at  the  start  of 

the  acquisition  cycle  in  order  to  obtain  R&M  improvements. 

The  most  reported  means  to  improve  R&M  focused  on  contracts, 

incentives,  and  valid  requirements.  The  thoughts  of  five 

managers  on  improving  R&M  can  be  understood  by  reviewing  one 

manager's  comments: 

The  biggest  benefit  and  most  cost  effective 
approach  to  improved  R&M  is  in  obtaining  detailed 
user  requirements  for  R&M.  Product  specifications 
. . .  that  detail  what  has  to  be  produced  coupled 
with  a  contract  to  enforce  compliance  with  the 
specifications  on  the  contractor. 

Several  managers  reported  that  the  use  of  incentives 

possibly  in  the  form  of  warranties  was  helpful  in  improving 

R&M  in  their  program.  Besides  better  specifications  and 

contractor  incentives,  proper  planning  and  execution  of 

reliability  growth  testing  and  the  use  of  a  TAAF  phase 

during  development  will  help  identify  potential  system 

problems.  These  techniques  can  also  expedite  system 

maturity.  In  one  program  by  working  producibi 1 ity  issues  to 

reduce  costs  and  improve  quality,  R&M  also  benefited. 

Another  area  was  mentioned  that  involved  working  with  the 

system  after  deployment.  Two  managers  recommended  the  need 
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to  keep  design  engineers  and  some  development  people  on  the 
system  through  the  maturation  phase  to  work  system  problems 
that  occur  after  deployment. 

Summary 

This  chapter  provided  the  results  of  the  thesis  data 
collection  effort.  Based  on  the  literature  review  findings 
successful  concurrent  programs  were  identified.  Some  of 
these  were  the  Thor,  Titan,  and  Polaris  ballistic  missiles. 
The  use  of  concurrency  was  shown  to  be  periodically  accepted 
and  rejected  because  of  acquisition  program  problems  with 
cost  and  system  performance.  The  primary  benefit  of  concur¬ 
rency  is  a  shortened  acquisition  cycle;  however,  this  also 
increases  program  risks  and  creates  problems.  Some  of  the 
potential  problems  in  concurrent  programs  are  poor  perform¬ 
ance  and  reliability.  The  low  correlation  between  testing 
and  field  R&M  measures  was  shown  to  be  a  problem  caused  by 
several  factors  which  included  the  R&M  definitions  and 
calculations. 

The  data  collected  from  the  interviews  was  presented 
and  showed  that  the  managers  reported  some  of  the  same 
information  as  the  literature.  The  acquisition  managers 
reported  the  primary  benefit  of  concurrency  as  a  shorter 
schedule.  This  benefit  is  also  a  major  element  in  the 
primary  problem  of  concurrency,  system  retrofit.  Eight 
hypotheses  were  evaluated  for  different  factors  that  could 
potentially  affect  system  R&M.  This  evaluation  was 
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performed  for  the  individual  managers '  responses  and  by 
program.  Additional  information  was  collected  on  factors 
which  the  managers  felt  impacted  system  R&M.  Through  an 
interview  question  the  managers '  views  on  how  to  improve 
system  R&M  showed  better  contract  specifications  and 
contractor  incentives  as  the  most  frequently  provided 
response.  The  conclusions  and  recommendations  based  on  this 
thesis  data  base  are  contained  in  the  next  chapter. 
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V.  Conclusions  and  Recommendat ions 


Introduction 

The  conclusions  and  recommendations  presented  in  this 
chapter  are  based  on  the  researcher's  findings  from  the 
literature  review  and  interviews  of  acquisition  managers 
involved  with  concurrent  acquisition  programs. 

Concurrency  is  an  acquisition  philosophy  that  has  been 
incorporated  by  DOD  since  its  first  military  use  on  the  ICBM 
program.  When  defined  as  an  acquisition  strategy,  concur¬ 
rency  is  an  overlap  of  some  or  all  of  the  acquisition 
process  phases.  These  phases  are  concept  exploration, 
demonstration  and  validation,  full  scale  development  (FSD), 
and  production/deployment.  The  use  of  concurrent  acquisi¬ 
tion  programs  has  been  debated  for  the  last  30  years.  The 
research  indicates  that  all  major  acquisition  programs  use 
some  degree  of  concurrency.  As  demonstrated  by  all  the 
weapon  system  programs  used  in  this  study,  the  phases  most 
frequently  overlapped  are  FSD  and  production/deployment. 

Over  the  years  many  successful  weapon  systems  have  been 
developed  using  concurrency.  Some  of  these  systems  are  the 
Atlas,  Titan,  Thor,  and  Minuteman  ICBMs.  Also,  several  air¬ 
craft  programs  successfully  used  concurrency  and  produced 
effective  systems  including  the  U-2,  SR-71,  F-15,  F-5E,  and 
F-16C/D.  Despite  these  numerous  successes  several  concur¬ 
rent  acquisition  programs  have  been  considered  failures. 


Concurrency  Benef its/Problems 


The  reasons  tor  the  continued  discussions  about  concur¬ 
rency  can  be  found  in  the  reported  benefits  and  problems. 

Concurrency  is  used  primarily  to  shorten  the  acquisi¬ 
tion  cycle  and  reduce  acquisition  costs.  Concurrency  is 
usually  planned  from  the  beginning  of  the  acquisition  phase. 
It  has  also  been  used  to  meet  firm  IOC  dates  once  a  program 
schedule  has  slipped  due  to  technical  and/or  funding 
problems.  Efforts  to  reduce  the  length  of  time  to  acquire 
new  weapon  systems  is  also  the  cause  of  many  of  the  problems 
associated  with  concurrency.  Acquisition  programs  with 
optimistic  schedules  which  experience  technical  problems 
and/or  a  slip  in  the  availability  of  test  articles  result  in 
concurrency  related  problems.  Some  of  these  problems  are 
inadequate  test  data  for  program  decisions,  system  produc¬ 
tion  and  deployment  with  incomplete  DT&E/IOT&E  testing, 
system  retrofits,  poor  performance,  use  of  interim 
contractor  support  (ICS),  and  low  system  R&M. 

A  major  concern  of  acquisition  managers  of  concurrent 
programs  is  the  need  for  system  retrofits  to  improve  system 
performance  and/or  R&M.  The  requirement  for  retrofit  of 
production  articles  and  the  use  of  ICS  may  negate  any  cost 
savings  produced  by  concurrent  acquisition.  The  end  result 
of  a  concurrent  acquisition  program,  success  or  failure, 
depends  on  the  decisions  and  skills  of  the  program  managers. 
As  previously  mentioned,  there  have  been  several  successful 


76 


and  unsuccessful  concurrent  programs.  However,  the  scope  of 
this  research  will  not  allow  further  comment  in  the  area  of 
overall  program  outcomes.  This  study  is  limited  to 
reviewing  the  impact  of  concurrency  on  system  R&M. 

Concurrency 's  Impact  on  R&M 

The  proven  techniques  for  developing  R&M  such  as  relia¬ 
bility  growth  testing,  parts  derating,  burn-in,  and  a  TAAF 
phase  increase  the  development  time  and  cost  of  acquisition 
programs.  Consequently,  the  primary  benefits  of  concur¬ 
rency,  a  shorter  schedule  and  lower  acquisition  cost,  are  in 
direct  conflict  with  improving  weapon  system  R&M.  Also,  a 
shorter  acquisition  cycle  provides  less  time  to  fully 
develop  system  performance.  Since  system  performance  is 
still  the  primary  element  in  obtaining  funding  and  ensuring 
program  continuation,  acquisition  managers,  senior  Air  Force 
leadership,  and  Congress  focus  on  this  element.  Therefore, 
system  R&M  development  receives  secondary  consideration. 
Concurrency  also  impacts  R&M  through  its  impact  on  other 
factors. 

Other  Factors  Affecting  R&M 

Concurrency  impacts  the  factors  in  this  section  by 
reducing  the  time  to  develop  and  accomplish  them.  The 
research  shows  that  the  existence  and  impact  of  the 
following  factors  is  dependent  on  the  individual  acquisition 
program. 
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Contractor  Personne 1 ■  Today  's  acquisition 


programs  use  contractor  maintenance  during  system  develop¬ 
ment.  The  use  of  contractor  personnel  to  perform  system 
maintenance  during  DT&E  and  sometimes  during  IOT&E  results 
in  lower  system  field  R&M  measures  than  the  R&M  measures 
demonstrated  during  testing.  This  reduction  in  field  R&M  is 
the  result  of  the  lower  skill  and  experience  levels  of  the 
user  command  maintenance  technicians.  The  negative  impact 
to  R&M  of  this  factor  will  be  eliminated  as  user  personnel 
gain  experience  on  the  weapon  system.  Acquisition  programs 
for  weapon  systems,  like  missiles,  which  have  little  organi¬ 
zational  and  intermediate  level  maintenance  will  not  experi¬ 
ence  any  negative  R&M  impact  from  this  factor.  SPO  and 
AFOTEC  acquisition  managers  consider  the  use  of  contractor 
maintenance  as  a  factor  which  influences  R&M  while  DPML 
personnel  feel  there  is  no  problem. 

Support/Test  Equipment.  Concurrent  acquisition 
programs  do  not  have  support  or  test  equipment,  which  is 
projected  for  field  use,  available  for  DT&E/IOT&E.  This 
research  produced  inconclusive  data  as  to  the  impact  on  R&M 
measures  in  all  programs.  The  data  indicates  that  the 
potential  exists  for  negative  impacts  to  field  R&M  measures. 
The  interviewees  did  not  agree  on  the  potential  R&M  impact. 

Tech  Data.  In  general  concurrent  acquisition 
programs  will  not  have  Air  Force  type  tech  data  available 
during  testing.  This  is  especially  true  if  contractor 
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personnel  are  performing  system  maintenance.  SPO  and  DPML 
personnel  do  not  feel  that  a  lack  of  tech  data  during  DT&E/ 
IOT&E  will  reduce  the  accuracy  of  R&M  measures.  However, 
AFOTEC  personnel  disagree  with  this  assessment. 

DT&E/ IOT&E.  Concurrent  aircraft  modification 
programs  have  incomplete  DT&E/IOT&E  prior  to  the  production 
decision.  This  factor  reduces  system  field  R&M  measures 
when  compared  to  testing  demonstrated  measures.  In  some 
cases  acquisition  programs  involving  modifications  will  have 
a  production  decision  before  the  start  of  DT&E/IOT&E.  This 
study  showed  that  the  acquisition  programs  developing  a 
missile  and  pod  completed  planned  DT&E  and  IOT&E  prior  to 
the  production  decision.  However,  additional  DT&E/IOT&E  was 
needed  to  evaluate  fixes  for  problems  found  during  the 
initial  testing  phases. 

Design  Freeze.  While  some  concurrent  acquisition 
programs  have  a  design  freeze  for  IOT&E,  there  is  no  impact 
on  system  R&M.  In  today 's  acquisition  environment  a  system 
design  freeze  depends  on  the  individual  acquisition  program. 

ECPs.  Concurrent  acquisition  programs  have  open/ 
untested  ECPs  at  the  time  of  the  production  decision.  While 
AFOTEC  personnel  feel  that  open/ untested  ECPs  will  reduce 
the  accuracy  of  R&M  measures,  SPO  and  DPML  managers  do  not 
share  this  concern. 

TAAF.  When  an  acquisition  program  has  an  incom¬ 
plete  TAAF  phase,  the  weapon  system  will  have  lower  system 
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R&M  measures  in  the  field  than  the  testing  demonstrated 
measures.  The  non-aircraft  programs  used  in  this  study 
showed  that  they  will  complete  their  TAAF  phases  more  often 
than  aircraft  programs. 

Other  Issues  Af f ect ing  R&M.  An  important  factor 
which  influences  a  weapon  system's  R&M  and  was  not  covered 
by  the  thesis  hypotheses  is  the  calculation  of  R&M  measures. 
The  accurate  determination  of  system  field  R&M  is  important 
in  that  it  affects  numerous  real  world  issues.  R&M  measures 
are  used  to  determine,  among  other  items,  spares  procurement 
and  unit  manning  posture.  Currently,  there  is  a  problem  in 
correlating  the  testing  predicted  R&M  measures  to  the  field 
results.  Many  plausible  explanations  exist  for  the 
disparity  in  R&M  measures.  The  main  issue  is  the  differ¬ 
ences  in  understanding  what  and  how  R&M  is  measured  in  the 
developmental  and  operational  environment.  During  develop¬ 
ment  reliability  is  presented  as  an  MTBF  derived  from 
chargeable  failures  and  system  operating  time.  While 
operational  reliability  is  calculated  as  an  MTBMA  which  uses 
the  number  of  system  failures  and  flying  time.  A  system 
MTBF  is  sometimes  used  in  the  field.  This  MTBF  measure 
removes  some  of  the  failures  used  to  calculate  MTBMA,  but  is 
usually  still  lower  than  the  testing  demonstrated  MTBF. 

Most  acquisition  managers  feel  that  R&M  goals  have  been 
contractually  met  and  through  system  maturity  the  predicted 
R&M  measures  will  be  achieved  in  the  field.  The  literature 
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shows  that  the  required  system  field  R&M  measures  are  not 
being  obtained.  This  real  world  problem  which  impacts 
system  availability,  combat  capability,  and  numerous 
logistics  areas  needs  to  be  corrected. 

Another  issue  which  could  contribute  to  the  problem  of 
low  system  R&M  was  identified  during  this  study.  Acquisi¬ 
tion  managers  in  the  same  program  and  managers  in  general 
did  not  agree  on  the  impact  of  the  hypothesis  factors  on 
system  R&M.  This  disagreement  will  result  in  some  factors 
which  affect  R&M  not  being  worked. 

Recommendat ions  for  Better  R&M 

While  certain  development  and  testing  techniques  can 
improve  system  R&M,  the  Air  Force  must  first  develop  a 
standardized  set  of  definitions  and  calculations  to  deter¬ 
mine  R&M  measures.  The  importance  of  R&M  to  system  avail¬ 
ability  and  combat  capability  makes  it  imperative  that 
testing  demonstrated  system  R&M  be  equal  to  the  field  R&M 
results.  During  system  development  and  testing  ail  failures 
that  occur  and  could  occur  in  the  operational  environment 
should  be  included  in  the  R&M  calculations.  Repetitive 
failures  should  not  be  eliminated  from  R&M  calculations 
because  a  proposed  fix  is  not  developed  and  incorporated  in 
the  system  at  the  time  of  occurrence.  Most  fixes  only 
extend  the  time  between  failures,  they  do  not  eliminate 
them.  The  definitions  and  calculations  for  contracted  R&M 
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measures  should  be  established  so  that  they  duplicate  those 
used  in  the  operational  environment. 


Once  accurate  R&M  definitions  and  calculations  repre¬ 
senting  the  field  environment  are  developed.  They  must  be 
placed  in  acquisition  contracts  to  ensure  that  contractors 
are  aware  of  the  R&M  requirements.  The  Key  elements  needed 
to  produce  the  high  levels  of  R&M  required  in  weapon  systems 
are  senior  management  support  for  R&M,  time  to  test  and 
develop  R&M,  and  the  funding  to  obtain  R&M. 

System  performance  will  continue  to  be  the  initial 
driving  element  in  weapon  system  acquisitions.  Therefore, 
program  schedules  -  may  require  additional  time  to  test  and 
evaluate  system  R&M.  The  inclusion  of  a  TAAF  phase  in  each 
acquisition  program  will  improve  system  R&M  and  will  help 
the  system  reach  maturity  early  in  its  operational  life 

Recommendat ions  for  Additional  Research 

The  following  recommendations  are  suggested  for  future 
research.  This  study  showed  that  several  acquisition 
managers  concurred  with  the  hypotheses.  Also  besides  the 
hypothesis  factors,  other  factors  were  identified  which  may 
negatively  impact  R&M.  Additional  research  is  needed  to 
expand  the  data  base.  A  study  to  determine  the  opinions  of 
additional  acquisition  managers  is  one  possibility. 

Several  differences  existed  in  the  factor's  status  and 
impact  to  R&M  between  the  aircraft  and  non-aircraft  programs 
used  in  this  study.  Recommend  additional  research  on  the 
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structure  and  flow  of  the  acquisition  program  plans  to 
determine  the  reasons  for  the  differences. 


Another  recommendat ion  for  study  is  to  look  at  the 
differences  in  R&M  definitions  used  in  the  developmental  and 
operational  environments.  This  study  should  focus  on  devel¬ 
oping  new  contractual  definitions  that  represent  user 
requirements  for  R&M,  possibly  as  an  MTBMA. 

Since  one  of  the  primary  benefits  of  concurrent  acqui¬ 
sition  is  cost  savings  and  a  major  problem  is  system 
retrofit,  this  area  warrants  investigation.  Retrofit  of 
production  articles  can  be  costly  and  these  may  eliminate 
any  development  savings.  Recommend  a  study  to  look  at  the 
cost  of  system  retrofits  in  relation  to  the  projected 
savings  of  concurrency- 

The  results  of  this  thesis  show  that  the  problems  of 
improving  R&M  are  affected  by  concurrency  but  only  through 
other  factors.  Continued  study  to  find  ways  to  control  the 
factors  will  improve  system  acquisition.  This  study  is  3 
start . 
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Appendix  A:  Personnel  Interviewed 


Beavers,  Lt  Col  Jeffery  C. 


Bock,  Maj  Larry  K. 
Eckert,  Col  John  S. 


Gillette,  Capt  David 


Test  Manager  Advanced  Systems 
Headquarters  AFOTEC 

LANTIRN  Acting  DP ML 

Deputy  System  Program  Director 
Deputy  for  B-1B 

F-15  Integrated  Logistics 
Support  Operations  Branch 
Chief 


Harland,  Mr.  Billy  C. 


Hiatchi,  Col  Melvin 


Horner,  Col  John  C. 


Jordon,  Lt  Col  Larry  M. 


Deputy  Director,  Strike  SPO 
Deputy  for  Reconnaissance/ Strike 
and  Electronic  Warfare  Systems 

Director  of  Integration  and 
Tests,  F-16 

Director  of  Logistics/Deputy 
Director  for  Logistics,  F-16 

Director  of  Operations  B-l  FOT&E 
Test  Team 


MacDonald,  Lt  Col  Arthur  S.  F-16  LANTIRN  OT&E  Director 


Miranda,  Col  Frederick  J. 


Monaghan,  Maj  Jeffery  C. 
Schnick,  Maj  Robert  H. 
Shearer,  Col  Richard 


Director  of  Acquisition 
Logistics,  B-1B 

Deputy  for  Operations  F-15E  OT&E 

AFOTEC  F-16  MSIP  Test  Director 

Director  Maverick  System  Program 
Of  f ice 


Stephens,  Maj  Boyd 
Tuck,  Mr.  Frank 


Assistant  DPML  for  Maverick 

F-15  Deputy  System  Program 
Director  Deputy  for  Tactical 
Systems 
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Appendix  B:  Interview  Question  1_ 

The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


What  is  your  general  assessment  of  the  use  of  concurrency  in 
acquisition? 


Manager  1 

The  goodness  or  badness  of  it  [concurrency 3 ...  wi 1 1  have  to 
wait  to  see  the  impact  when  we  meet  IOC.  Other  factors  will 
overcome  some  concurrency  problems. 

Manager  2 

A  generic  response.  The  book  says  your  not  supposed  to  do 
it  but  the  real  world  says  you  have  to  do  it.  It's  a  fact 
of  life.  Concurrency  is  not  altogether  bad  but  it  is  high 
risk. 

It  was  appropriate  for  this  program  since  it  was  not  a  new 
state  of  the  art.  Some  concurrency  worked  on  this  program 
and  saved  money. 


Manager  3 

Not  a  very  smart  idea.  We  should  not  be  forced  into  concur¬ 
rency.  However,  the  way  the  real  world  is  we  must  use  it. 

By  having  proper  cooperation  between  the  developers  and 
testers,  concurrency  can  be  made  to  work. 

Manager  4 

Concurrency  cuts  down  on  system  lead  time  but  makes  manage¬ 
ment  of  the  program  difficult. 

Manager  5 

Generally,  I  think  it's  almost  mandatory  that  you  have  some 
level  of  concurrency  with  zero  concurrency  you  would  never 
get  anything  done.  You  need  some  level  of  concurrency.  In 
the  long  run  concurrency  is  going  to  help  keep  down  cost  and 
get  systems  into  the  field  earlier. 

You  buy  some  problems  with  concurrency.  Things  will  get  out 
of  sync  and  you'll  be  producing  hardware  and  not  be  able  to 
get  all  the  changes  into  it  [system!. 
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Manager  6 


It  ' s  a  way  of  life.  It  's  something  we  have  to  do  to  save 
money  but  there  is  a  cost  of  doing  business  with  concur¬ 
rency.  You  end  up  with  a  lot  of  retrofit  in  the  field. 

Manager  7 

Having  been  on  several  programs  one  of  which  was  non-concur¬ 
rent,  concurrency  in  today 's  environment  is  almost  a  neces¬ 
sity  because  of  the  resources  and  monetary  values  involved 
with  new  weapon  systems. 


Manager  8 

Concurrency  should  be  avoided  whenever  possible.  In  our 
program  the  capability  to  build  the  aircraft  was  there  but 
logistics  had  been  deferred  and  had  to  be  developed.  This 
created  a  lot  of  configuration  problems. 

[Concurrency!  may  result  in  a  large  amount  of  interim 
contractor  support. 

Design  changes  of  the  system  generated  by  new  user  require¬ 
ments  are  difficult  to  incorporate  and  this  problem  is 
compounded  by  concurrency. 


Manager  9 

I  think  for  most  of  the  programs  we  use  concurrency  most  of 
the  time.  The  primary  reason  for  it  being  to  shorten  the 
schedule  for  acquisition. 


Manager  10 

I  think  it  [concurrency!  is  necessary.  It  has  some 
disadvantages  but  I  also  think  it  has  some  advantages.  It 's 
about  the  only  way  in  the  real  world  that  you  can  do  it 
[acquisition! . 

You  don't  have  any  choice  you  use  it  mainly  to  stay  in  some 
sort  of  budget  and  schedule.  You  might  start  out  with  no 
concurrency  but  before  you  know  it  you  will  have  concur¬ 
rency. 


Manager  11 

I  think  it  is  best  for  some  programs  and  inappropriate  for 
others  so  it  must  be  looked  at  carefully.  Concurrency's 
success  depends  on  the  quality  of  management  in  the  SPO. 
They  must  be  able  to  anticipate  the  problems  and  get  the 
contractor  looking  at  them  [problems!  before  they  happen. 


Manager  12 


From  the  way  our  programs  are  going  now  and  the  length  of 
time  it  takes  to  develop  systems,  to  get  anything  fielded 
now  it  is  almost  inevitable  that  you  are  going  to  have 
concurrency. 

So  I  don't  necessarily  see  it  as  bad. 

Manager  13 

It's  the  only  way  to  go,  if  we're  going  to  keep  the  acquisi¬ 
tion  process  going  and  keep  pace  with  technology.  Technol¬ 
ogy  is  moving  so  fast  that  if  you  don't  use  concurrency  the 
user  can't  get  what  he  needs  in  the  system  in  terms  of 
technology. 

The  type  of  program  has  to  be  looked  at  when  you  consider 
using  concurrency. 

Management  and  coordination  of  the  program  is  very  important 
in  making  concurrency  work. 

Manager  14 

Concurrency  has  helped  my  program-  It  is  important  in  that 
it  provides  the  ability  to  identify  operational  [field! 
problems  early. 


Manager  15 

Concurrency  is  neither  good  or  bad  it  depends  on  the  system. 
With  a  low  risk/ techno  logy  system  the  use  of  concurrency 
should  not  cause  any  problems.  A  high  risk/new  technology 
system  should  not  use  concurrency.  However,  you  must  look 
at  how  rapidly  the  technology  is  changing.  Using  a 
non-concurrent  program  in  a  rapidly  changing  technology 
system  may  result  in  deployment  of  an  obsolete  system. 

I  don't  see  concurrency  as  bad. 
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Appendix  C :  Interview  Question  1_ 


The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


Who  was  responsible  for  the  maintenance  of  your  system 
during  DT&E  and  IOT&E  and  at  what  level?  (organizational, 
intermediate,  depot)  If  the  contractor,  does  this  bias  the 
R&M  data? 


Manager  1 

The  contractor  was  responsible  for  all  three  levels. 

I  don't  think  this  biased  R&M  data  because  we  have  Air  Force 
blue  suiters  there.  We  have  JRMET,  AFOTEC  representatives, 
and  Test  Force  personnel  to  overview  the  data. 

Manager  2 

Largely  contractor  supported. 

Using  contractor  personnel  could  be  a  factor  in  causing 
biased  R&M  on  a  new  system,  but  in  our  case  a  new  version  of 
the  current  system  it  did  not  affect  it  (R&M  data!. 

Manager  3 

During  DT&E  maintenance  was  performed  by  the  contractor. 

For  IOT&E  we  had  some  contractor  support  and  transitioned  to 
blue  suit  maintenance. 

There  was  no  bias  in  this  program's  R&M. 

Manager  4 

In  the  DT&E  phase  the  contractor  did  maintenance  on  the  new 
equipment  with  crew  chief  assistance.  OT&E  has  some 
contractor  support  with  mostly  blue  suit  maintenance. 

Contractor  personnel  may  bias  data. 

Manager  5 

During  test  and  development  the  contractor  was  responsible 
for  maintenance.  Our  system  has  very  little  maintenance  in 
the  field. 
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I  don't  think  contractor  maintenance  biased  R&M  data.  We're 
seeing  better  1R&M1  in  the  field  than  the  contract  reported. 
This  is  an  unusual  circumstance. 

Manager  6 

In  DT&E  the  contractor  performed  all  levels  of  maintenance. 

I  have  no  evidence  of  bias. 

Manager  7 

Depending  on  the  type  of  testing  done  on  the  aircraft,  at 
the  organizational  level  some  systems  were  maintained  by  the 
contractor  and  some  by  the  Air  Force.  Intermediate  level 
was  done  by  the  contractor. 

Does  not  result  in  bias. 


Manager  8 

A  combination  of  contractor  and  Air  Force  blue  suit  at  the 
0- level.  At  the  I- level  it  was  interim  contractor  support. 

Yes,  it  can  result  in  biased  R&M  because  of  skill  differ¬ 
ences. 

Manager  9 

We  had  contractor  maintenance  on  some  systems  and  as  much  as 
we  could  we  tried  to  have  Air  Force  at  least  over  the 
shoulder  maintenance,  we're  talking  about  0- level.  For 
I- level  most  of  the  maintenance  was  contractor. 

There  may  be  some  limited  bias  since  we  used  both  contractor 
and  Air  Force  personnel. 


Manager  10 

The  contractor  at  the  0- level,  the  I- level,  and  depot  level. 

The  contractor  has  to  bias  it  CR&M  data!  a  little  because  he 
has  highly  qualified  people  compared  to  the  average  GI.  He 
most  likely  has  higher  paid  more  experienced  people  and  that 
would  make  the  reliability  and  maintainability  picture  a 
little  brighter. 
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Manager  11 


The  contractor  totally. 

Yes,  using  the  contractor  biases  R&M.  Also,  data  from  OT&E 
is  biased  because  of  testing  on  FSD  items. 

Manager  12 

At  the  test  sites  it  will  be  the  contractor.  We  either  have 
ICS  or  support  contracts  where  the  contractor  comes  in  and 
as  part  of  his  development  program  shows  that  his  support 
equipment  is  actually  doing  the  job.  We're  also  trying  to 
become  organic. 

Yes,  it  biases  the  maintainability  data  more  than 
reliability  data.  Your  going  to  have  a  failure  regardless 
of  who  fixes  it,  but  then  you  might  not  necessarily  see  some 
of  the  problems  with  your  support  equipment  or  some  of  the 
problems  with  training.  Your  equipment  will  tend  to  get 
fixed  quicker  than  in  the  real  world. 

Manager  13 

We  had  heavy  contractor  involvement  in  DT&E.  Using  command 
and  Systems  Command  blue  suiters  were  also  used  to  perform 
maintenance  and  evaluations  of  the  system  during  our 
combined  DT&E  and  IOT&E  phases. 

I 'm  not  sure  that  using  contractor  personnel  biased  the  R&M 
data.  It  could  in  some  ways  but  it  is  not  big  since  very 
little  R&M  data  comes  from  DT&E.  IOT&E  data  is  more 
important  in  evaluating  and  proving  R&M. 

Manager  14 

We  had  two  groups  performing  maintenance  on  our  system  both 
the  contractor  and  Air  Force.  The  determination  as  to  which 
group  worked  which  system  was  dependent  on  the  type  of 
testing  and  the  level  of  modification.  During  IOT&E  only 
Air  Force  personnel  were  used. 


There  may  be  some  bias. 


Manager  15 


We  used  both  contractor  and  blue  suit  maintenance. 

Using  contractor  personnel  may  bias  R&M  data  but  only  in  the 
initial  deployment  stage  because  the  Air  Force  personnel 
have  not  had  the  chance  to  build  up  the  same  system 
experience  as  the  contractor. 

As  the  system  reaches  maturity  and  the  blue  suiters  get  more 
experience  the  R&M  predictions  Should  be  met. 
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Appendix  D:  Interview  Question  14 

The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


Did  concurrency  reduce  your  ability  to  use  a  test,  analyze, 
and  fix  procedure  to  identify  potential  reliability 
problems? 

Manager  1 

No,  I  don't  think  so.  We  had  problems  in  the  fact  that  we 
didn't  have  enough  assets  in  flight  test  that  we  could  sit 
there  and  say  OK  we  had  a  fault,  go  out  and  test  it,  come  up 
with  a  fix,  and  institute  it  back  in.  Of  all  the  items  we 
had  identified  in  flight  test  and  development  that  needed  to 
be  fixed  Ctheyl  weren't  going  to  be  implemented  until 
production,  so  that  was  an  issue.  That  wasn't  a  problem  of 
concurrency.  We  had  limited  assets  during  development. 

Manager  2 

No.  The  procedure  has  continued  to  be  used  to  further 
enhance  the  system. 

Manager  3 

Yes,  it  Impacted  TAAF.  There  were  some  technical  problems 
experienced  and  this  resulted  in  IOT&E,  DT&E,  and  maturity 
problems.  Some  fixes  had  to  be  checked  in  IOT&E  not  DT&E. 

Manager  4 

Concurrency  has  not  caused  any  problems  for  TAAF.  We  will 
coordinate  DT&E  and  IOT&E  requirements  to  minimize  the 
potential  impacts. 

Manager  5 

I  can  relate  that  CTAAF1  to  concurrency  but  I'll  give  you 
another  thought  on  the  matter.  TAAF  is  an  excel lant  tool 
during  development  testing.  Unfortunately  ,our  system  is  so 
politicized  that  the  risk  of  a  failure  has  become  astronom¬ 
ical.  TAAF  maybe  the  cheapest  way  to  find  out  what's  wrong 
and  to  go  ahead  and  fix  it.  If  you  have  failures  puople 
will  talk  about  cancelling  your  program  and  have  a  memory 
that  things  are  terrible. 
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Manager  6 


Yes,  definitely  because  by  the  time  you  figure  out  what  to 
test,  analyze,  and  fix  you  have  already  built  in  a  delay 
where  you  have  units  in  the  field  that  are  unmodified  with 
the  fixes.  So  you  either  have  to  come  back  and  do  a 
retrofit  or  live  with  2  or  3  different  configurations  which 
causes  problems  in  configuration  management  in  the  life 
cycle  of  the  system. 


Manager  7 

We  didn't  use  TAAF  generally  on  black  boxes.  We  used  devel¬ 
opmental  reliability  qual  tests,  maintainability  demos,  and 
production  reliability  demos.  Those  specifications  were  a 
lot  harder  than  TAAF  to  get  through,  especially  for  tests 
like  the  production  reliability  qual  tests.  If  you  found  a 
deficiency  in  a  box  the  contractor  had  to  bring  all  the 
boxes  already  produced  up  to  that  point.  This  was  at  the 
contractor's  expense-  There  were  lab  threshholds  that  had 
to  be  met. 


Manager  8 

We  used  a  lot  of  test,  analyze,  and  fix  on  individual 
systems  but  this  was  in  the  lab-  We  could  have  used  more 
CTAAF1  if  we  had  more  time  to  test  before  we  went  into 
production.  The  test,  analyze,  and  fix  we  did  do  was  very 
ef  f  ect ive. 

Manager  9 

I  don't  think  it  did.  Only  for  some  of  the  systems  did  we 
use  a  test,  analyze,  and  fix  technique.  When  we  did,  it  did 
not  appear  that  concurrency  affected  that  method- 

Manager  10 

I  don't  think  so-  We  did  a  lot  of  test,  analyze,  and  fix. 

I  better  qualify  that  a  little  bit,  some  of  that  CTAAFJ  was 
for  reliability  and  some  was  for  improving  performance. 
Concurrency  didn't  cause  us  not  to  be  able  to  do  TAAF. 

Manager  11 

Yes,  primarily  in  only  one  part  of  the  system.  Some 
problems  identified  in  testing  and  fixed  could  not  be 
corrected  in  the  test  articles  and  retested  before 
production. 


TJl 
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Manager  12 


No,  but  what  your  going  to  find  is  we're  out  there  flying 
equipment  while  we're  still  in  the  lab  doing  reliability 
growth  programs.  So  we're  still  using  it  CTAAF1  but  it's 
just  a  concurrent  phase.  We  have  lab  testing  going  on  at 
the  same  time  we  have  operational  testing  going  on. 

Manager  13 

No,  I  think  we  did  a  lot  in  R&M  and  operational  areas.  I 
think  there  may  be  some  reluctance  to  make  some  of  the 
changes  because  of  cost. 


Manager  14 

Yes,  the  schedule  only  allowed  a  short  analyze  period.  This 
made  it  hard  to  really  evaluate  the  full  system. 

Manager  15 

We  didn't  use  TAAF  on  the  whole  system.  There  was  some  work 
for  reliability  on  the  subsystems  with  no  problems  caused  by 
concurrency. 
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Appendix  E:  Interview  Question  16 


The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


What  are  the  primary  benefits  of  a  concurrent  program? 

Manager  1 

Concurrency  allows  you  to  meet  a  schedule.  If  you  have 
problems  in  development  and  I'm  not  saying  just  development 
problems,  but  in  our  case  funding  problems  that  cause  you  to 
stretch  out  development,  by  having  concurrency  you  can  still 
meet  that  end  item  schedule,  IOC.  It  allows  you  to  absorb 
some  program  problems  upfront  and  still  meet  a  delivery 
date. 


Manager  2 

Schedule  a  very  real  fact  of  life  in  any  program,  a  shorter 
schedule.  By  saving  years  on  the  front  end  of  a  program  it 
is  easier  to  get  funding  even  though  you  put  the  assurance 
that  R&M  is  correct  at  risk. 

Manager  3 

A  distinct  benefit  is  getting  operational  expertise  involved 
early  in  the  program.  This  allows  fixes  to  be  geared  to 
field  operational  experience  and  need. 

Manager  4 

Concurrency  provides  a  shorter  schedule.  Gets  equipment  in 
the  field  faster  and  reduces  the  chance  of  program  cancel¬ 
lation.  You  can  always  go  back  and  fix  it  once  you  have  it. 

Manager  5 

Primarily  schedule  is  a  big  benefit.  If  you  did  anything 
sequentially  by  the  time  you  fielded  a  system  it  would  be 
overcome  by  technology  and  you  wouldn't  get  it  fielded.  The 
second  is  cost. 


Manager  6 

Saves  money  for  the  overall  program  and  shortens  the  devel¬ 
opment  cycle. 
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Manager  7 

Obviously  cost,  hopefully  schedule,  and  I  think  performance. 

Manager  8 

Your  able  to  field  weapon  systems  quicker  and  avoid 
political  turmoil  and  budget  problems. 

Manager  9 

The  primary  benefit  is  schedule. 

Manager  10 

Improves  schedule  and  the  acquisition  cost,  if  it 's  done 
smartly,  is  reduced.  It  saves  money  by  getting  the  Job  done 
and  over  with  instead  of  stretching  it  out. 

Manager  1 1 

Primarily,  it  shortens  the  acquisition  cycle  but  there  i3 
added  risk.  Good  program  management  is  needed  to  provide 
for  risk  reduction. 


Manager  12 

Getting  the  system  fielded  quicker,  in  a  shorter  period  of 
time. 


Manager  13 

You  get  the  system  out  to  the  user  earlier.  The  system  may 
not  have  its  full  capabilities  but  we  can  start  training  our 
people,  operations  and  maintenance,  and  begin  building  their 
experience  on  the  system. 

Manager  14 

You  get  the  user  involved  and  are  able  to  evaluate  the 
operational  capabilities  of  the  system  early.  So  when  you 
get  done  with  development  you  are  not  usually  surprised  by 
major  problems  in  the  field. 

Manager  15 

Using  concurrency  allows  you  to  get  a  system  in  the  field 
quicker  and  saves  money.  It  is  important  for  acquisition  of 
rapidly  changing  technologies. 
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Appendix  F :  Interview  Question  17 


The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


What  problems  are  associated  with  a  concurrent  program? 

Manager  1 

Trying  to  develop  and  produce  at  the  same  time  and  you  may 
not  have  the  final  item  developed  before  you  go  into  produc¬ 
tion.  In  our  program  . . .  concurrency  is  not  as  big  an 
issue.  The  first  few  production  items  are  being  used  for  TO 
development,  training,  and  used  in  house  for  reliability 
qual  testing.  The  first  production  units  to  the  field  are 
still  a  few  years  out,  we're  still  in  development.  This 
allows  us  to  incorporate  significant  developments  into  the 
first  units  going  to  the  field. 

Manager  2 

There  is  a  problem  in  assuring  that  R&M  is  going  to  be  what 
it  needs  to  be  for  the  user  in  the  field  as  opposed  to  what 
you  may  have  demonstrated  during  the  current  test  phases. 

We  had  problems  with  new  test  equipment  which  passed  all 
qualification  tests  in  the  contractor  facility.  However,  in 
the  field  it  had  very  poor  reliability. 

Manager  3 

If  concurrency  is  planned  from  the  beginning  it  can  only  be 
a  good  way  of  doing  business. 

When  you  are  driven/forced  into  it,  the  pressure  to  complete 
testing  to  meet  delivery  hinders  DT&E  personnel  in  accom¬ 
plishing  their  Job. 


Manager  4 

Concurrency  makes  a  program  difficult  to  manage.  It  is  hard 
to  ensure  that  everything  gets  done  and  is  moving  in  the 
right  direction. 


Manager  5 

There  are  problems  with  support  equipment. 

Mostly,  you'll  find  things  that  Just  don't  work.  It  will  be 
mandatory  to  go  back  and  retrofit  sometimes. 
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Systems  may  be  delivered  with  some  equipment  not  available 
at  the  time  of  delivery. 


Manager  6 

The  lack  of  the  ability  to  do  adequate  maintenance  planning 
early  enough  and  to  buy  support  equipment  and  spares  in 
adequate  quantities  to  support  the  first  deliveries. 

It  impacts  our  ability  to  get  reliability  testing  and 
maintainability  testing  accomplished  and  then  incorporate 
any  f ixes. .  . 

There  are  negative  impacts  on  virtually  every  ILS  element. 

Manager  7 

Extremely  tight  schedules  with  almost  no  leeway. 

Sometimes  there  is  a  degree  of  risk  associated  with  system 
performance. 

We're  constantly  slighly  late  getting  the  right  configura¬ 
tion  of  support  and  test  equipment  out  to  meet  the  system 
and  that  is  hampered  by  concurrency  because  we're  just 
barely  finishing  up  development  before  we  start  production. 

Manager  8 

The  biggest  problem  with  concurrency  in  our  program  was  that 
systems  were  fielded  before  support  equipment,  tech  data, 
and  engineering  data  were  available. 

Results  in  more  retrofits  of  the  system  after  production. 

Manager  9 

Problems  you  would  expect  that  things  don't  get  fixed 
correctly  before  production  decisions  are  made. 

Production  decisions  are  made  on  phase  designs  rather  than 
final  designs. 

There  is  a  shortening  of  the  test  and  evaluation  process. 

Manager  10 

Probably,  somewhat  of  an  unstable  configuration  that  your 
having  to  make  a  lot  of  changes  to,  some  of  those  can  be 
costly  but  I  still  think  overall  you  save  money.  I  think 
you  have  more  ECPs  and  the  program  is  a  little  more 
turbulent . 
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Generally,  you  could  end  up  with  a  design  that  could  not  do 
the  job  either  from  a  performance  or  R&M  standpoint. 

Manager  1 1 

Eventually,  you  pass  the  design  hardware  and  software  freeze 
points  during  testing.  After  that  point,  deficiencies  found 
and  fixed  cannot  be  implemented  into  the  production  articles 
wi thout  retrofits. 


Manager  12 

You  end  up  spending  a  whole  lot  of  downtime  retrofitting  the 
systems  that  you  have  to  modify  because  of  problems  you 
identified  in  the  test  program.  You  end  up  dedicating  a  lot 
of  your  people  and  resources  to  tracking  the  configuration 
of  boxes  that  are  totally  different  and  trying  to  structure 
some  turnaround  programs  to  get  the  latest  design  in  the 
field.  For  about  the  first  two  years  after  IOC,  our  whole 
dedication  is  going  to  be  trying  to  update  the  configuration 
of  the  boxes  that  were  fielded  before  we  got  to  IOC. 

Manager  13 

The  biggest  problem  is  management  of  the  concurrent  issues. 
It  is  difficult  to  ensure  the  proper  coordination  and 
distribution  of  limited  assets  to  accomplish  development, 
testing,  and  training.  Management  must  be  able  to  integrate 
all  the  functions  involved  in  the  acquisition  process 
engineering,  contractor,  testing  both  DT&E/IOT&E,  R&M,  and 
others. 


Manager  14 

The  manager  of  the  program  will  have  problems  in  dealing 
with  all  the  varying  inputs  and  in  deciding  when  it  is 
appropriate  to  bring  them  into  the  program.  This  carries 
over  into  selecting  the  proper  time  to  dedicate  resources 
and  at  what  issues.  Some  of  these  inputs  come  from  design 
requirements,  R&M  issues,  development  testing  and  evalu¬ 
ation,  operational  testing  and  evaluation,  and  may  include 
static  displays  and  system  orientation  visits. 
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Manager  15 


You  can’t  always  get  the  support/test  equipment  and  tech 
data  developed  and  out  to  the  field  with  the  system. 

There  will  probably  be  retrofits  required  to  fix  design 
deficiencies  and  to  bring  the  system  up  to  full  capability 
after  production  deliveries  start. 

It  takes  a  lot  more  people  to  do  it  simultaneously.  There 
are  a  lot  of  things  happening  at  once.  Requires  more 
manpower. 
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Appendix  G: 


Interview  Question  18 

The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


What  needs  to  be  done  to  improve  system  R&M  during  acquisi¬ 
tion? 


Manager  1 

They've  done  that  recently.  R&M  has  been  made  an  important 
area  of  program  management  but  until  it  is  impressed  on 
program  management  along  with  schedule  and  cost  it  won't  be 
an  issue.  Continued  emphasis  of  R&M  is  important. 

A  good  warranty  on  the  system  which  forces  the  contractor  to 
think  about  R&M  areas.  The  system  has  to  reach  the  required 
R&M,  otherwise  he  [contractor!  will  have  to  do  the  repairs. 

Manager  2 

The  testing  environment  must  duplicate  the  real  world  as 
much  as  possible.  In  our  program  a  test  set  met  the  mil 
standard  qualification  levels  and  tests  in  the  lab  but 
failed  to  reach  acceptable  performance  levels  in  the  field. 

Manager  3 

A  lot  of  things  are  being  done  from  R&M  2000  to  training  of 
acquisition  personnel.  Continued  emphasis  from  top  level 
management  about  the  seriousness  of  R&M  improvements. 

Manager  4 

The  biggest  benefit  and  most  cost  efficient  approach  to 
improved  R&M  is  in  obtaining  detailed  user  requirements  for 
R&M.  Product  specifications  must  be  written  that  detail 
what  has  to  be  produced  coupled  with  a  contract  to  enforce 
compliance  with  the  specifications  on  the  contractor. 

Manager  5 

As  you  approach  the  end  of  FSD,  you  shouid  plan  a  phase  to 
redesign  for  producibility  and  this  will  help  R&M.  In  our 
program  by  working  producibility  issues  to  reduce  production 
costs  and  improve  quality,  R&M  benefited. 

R&M  has  to  be  considered  early  in  the  program.  A  warranty 
seemed  to  help  on  our  program  by  driving  the  contractor  to 
improve  R&M. 
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Manager  6 


Find  some  way  to  give  the  contractor  an  incentive  from  the 
beginning  of  FSD  to  produce  a  reliable  product.  Coming  in 
at  a  later  stage  is  marginal  payback  at  best  and  results  in 
multiple  conf igurations. 

The  way  to  get  good  reliability  and  maintainability  is  to 
let  the  contractor  know  in  the  beginning  that  he  gets  a 
reward  or  a  penalty  for  producing  either  outstanding  R&M 
performance  or  deficient  R&M  performance.  Warranties  are 
one  way  to  do  this  but  not  the  only  way  to  provide 
incentives. 


Manager  7 

Besides  designed  in  objectives  when  you  first  get  the  weapon 
system  in  the  field  have  a  structured  system  established 
with  a  loop  back  to  the  design  and  test  engineers.  This  way 
for  any  initial  problems  with  a  could  not  duplicate  or  bench 
check  serviceable;  the  box,  test  set  in  AIS,  and  fault 
isolation  system  can  be  looked  at  to  fine  tune  the  system 
once  it  starts  performing  in  the  field. 

Manager  8 

Contractual  language  in  terms  of  reliability  and  maintain¬ 
ability  requirements  has  to  be  a  little  more  specific  and 
demanding  on  the  contractor. 

Manager  9 


I  think  the  key  thing  is  the  design  of  R&M.  Also,  relia¬ 
bility  growth  testing  and  test,  analyze,  and  fix  need  to  be 
properly  scheduled  and  planned. 


Manager  10 


One  thing  is  to  do  more  early  real  growth  testing  and 
earlier  maintainability  demos  by  bring  in  the  user  when  the 
prototypes  are  being  put  together.  This  would  be 
preliminary  testing  and  would  have  to  be  done  again  later. 

Another  thing  is  once  the  system  is  delivered  to  the  field, 
some  of  the  design  engineers  who  built  the  system  ought  to 
carry  it  through  the  maturation  phase  This  will  help 
identify  the  problems  that  crop  up  after  a  system  reaches 
the  field,  some  of  which  you  don't  see  in  development. 
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Manager  11 


Everyone  needs  to  think  about  R&M.  The  establishment  of  R&M 
2000  as  the  directive  to  improving  R&M.  The  operational 
side  in  Systems  Command  needs  to  be  educated  in  R&M. 

Manager  12 

More  money  has  to  be  allocated  for  that  purpose  [R&M! .  A 
lot  of  times  when  we  get  in  a  crunch  for  what  are  we  going 
to  spend  our  bucks  for,  it  is  easier  to  justify  changes  for 
performance  than  R&M. 

The  reliability  and  maintainability  improvements  are  going 
to  save  somebody  else  money  downstream.  Everybody  knows  it 
[R&M!  is  good  but  the  money  isn’t  there.  Maybe  you  need  to 
set  aside  some  logistics  money  and  not  let  anyone  else  into 
it.  Then  that  /ould  force  them  [logistics!  to  make  some  of 
the  decisions  on  what  we  can  afford  to  have  in  our  system. 

To  a  large  extent  it  is  a  mentality  problem.  You  want  the 
system  to  perform  first  and  then  you  can  work  the  problems 
of  how  to  fix  it  and  make  it  last  longer.  You  can  delay 
R&M.  You  can't  get  your  system  to  production  unless  you 
prove  it  performs.  The  whole  system  [acquisition!  is  geared 
that  way.  Your  program  will  be  killed  quicker  for  perform¬ 
ance  problems  than  for  R&M. 

Manager  14 

Get  some  knowledgeable  field  wise  people  in  to  look  at  and 
evaluate  the  system  early  in  the  program.  That  is  bring  in 
the  people  who  are  going  to  have  to  do  the  work  on  the 
system.  This  will  help  identify  potential  maintenance 
problems  in  time  to  get  them  corrected  before  the  system 
gets  to  the  field. 


Manager  15 

If  your  going  to  do  a  concurrent  program  you  might  want  to 
leave  out  some  risky  technology  or  have  a  parallel 
development  effort  [for  subsystems!  but  that  costs  money. 
That  is  have  two  developments  one  that  is  high  risk  and  one 
that  is  low  risk,  a  fall  back  in  case  the  high  risk/ 
technology  doesn't  work  out. 

Researcher:  What  techniques  such  as  reliability  growth 
testing  or  TAAF  can  we  spend  more  time  doing  that  will  help 
improve  R&M. 

I  don't  subscribe  to  the  theory  that  R&M  goes  down  when  you 
use  concurrency. 
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Appendix  H; 


Interview  Question  19 


The  following  information  was  taken  from  interviews. 
Bracketed  material  was  added  by  the  researcher. 


What  do  you  feel  accounts  for  the  differences  in  R&M 
measures  between  DT&E,  IOT&E,  and  field  results? 

Manager  1 

One  thing  is  that  in  DT&E  and  IOT&E  you  have  a  smaller 
number  of  systems.  While  in  the  field  you  have  more  systems 
doing  more  different  missions  than  in  test. 

You  develop  a  higher  level  of  maintenance  expertise  and 
knowledge  of  the  system  with  both  the  contractor's  personnel 
and  a  small  Air  Force  cadre.  In  the  field  you  have  a  lower 
maintenance  expertise. 

The  test  environment  is  benign.  The  field  support  equipment 
is  usually  different. 


Manager  2 

The  differences  between  the  lab  testing  environment  and  the 
real  world. 


Manager  3 

We  don't  have  uniform  definitions.  From  my  experience  on 
three  programs,  DT&E  and  operational  personnel  do  not 
interpret  definitions  the  same.  We  need  a  standard 
definition.  In  DT&E  we  use  MTBF  and  operational  needs  for 
reliability  are  expressed  as  an  MTBMA. 

Test  conditions  are  different  between  DT&E  and  IOT&E. 

Manager  4 

In  DT&E  and  IOT&E  the  Air  Force  personnel  used  are  very  good 
technicians.  This  combined  with  the  use  of  contractor 
personnel  gives  you  very  good  maintenance.  ,In  the  field  you 
have  lower  skilled  personnel. 
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Manager  6 


As  much  as  anything,  the  user's  measures  of  merit  are 
different  than  what  we  hold  the  contractor  responsible  for. 
There  is  a  difference  in  what  constitutes  a  failure.  There 
is  no  direct  translation  between  what  the  user  and  the 
contract  consider  important. 

Manager  7 

During  DT&E  you  have  a  very  select  group  that  knows  what 
their  doing.  The  emphasis  is  concentrated  on  a  few  systems. 
Comparing  that  performance  tduring  DT&E]  good  or  bad  to  what 
will  eventually  take  place  in  the  field  with  the  run  of  the 
mill  GI. . •  sometimes  there  is  no  relationship  between  them. 

IOT&E  and  OT&E  come  a  little  closer  Cto  the  field]  if  they 
are  not  concurrent  with  DT&E- 

Manager  8 

I  account  for  that  in  two  ways.  First,  you  have  experienced 
contractors  who  know  the  system  inside  and  out  and  have 
special  test  equipment. 

Second,  you  have  a  trained  group  of  blue  suit  maintenance 
personnel  who  are  better  than  the  normal  user  command 
personnel . 


Manager  9 

The  Air  Force  as  a  community  needs  to  standardize  the 
definitions  of  R&M.  In  many  cases  we  have  lab  values  which 
we  have  rules  of  thumb  for  translating  to  field  values  but  I 
think  we  need  to  standardize  the  definitions.  In  my  experi¬ 
ence  the  confusion  comes  about  when  you  want  to  have  an  R&M 
incentive  or  warranty  clause  in  the  contract.  You  have  to 
make  sure  that  you  and  the  contractor  really  understand  what 
numbers  to  use  and  to  measure  against. 

Manager  10 

Part  of  the  problem  comes  from  the  blue  suiters  in  their 
handling  of  the  system,  and  their  experience  in  trouble¬ 
shooting  and  maintaining  the  system. 

The  data  collection  early  on  with  the  contractor  and  SPO 
guys  is  more  likely  to  be  flavored  to  getting  the  system  to 
look  as  good  as  possible.  Let  us  say  you  have  a  failure  and 
you  don't  know  whether  It's  type  1  or  not.  We  use  JRMET  to 
analyze  and  figure  out  where  it  Cf allure]  ought  to  go  and  we 
sometimes  give  the  system  the  benefit  of  that.  We  get  rid 
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of  a  lot  of  faults  that  should  not  be  blamed  on  the  system 
your  testing.  You  don't  JRMET  failures  in  the  field. 

Where  as  later  on  the  GIs  are  just  filling  out  forms  to 
report  reliability.  They  are  not  out  to  give  any  credit  to 
a  given  system.  The  routineness  to  them  of  reporting  the 
reliability  would  have  an  impact.  The  actual  reliability 
may  be  as  much  in  the  field  but  is  preceived  to  be  poorer 
because  the  guy  didn't  do  the  same  type  of  [evaluation!  as 
the  guys  back  in  DT&E  would  do. 

Part  of  the  reason  could  be  the  definition  of  a  failure. 

The  environment  in  testing  is  not  really  representative  of 
the  operational  environment. 

Manager  11 

Differences  in  R&M  data  collection.  During  DT&E  info  is 
collected  by  the  contractor  and  can  be  incomplete  and 
biased.  Data  from  OT&E  is  collected  more  scientifically  and 
is  geared  to  determining  R&M  results.  The  normal  field 
system  is  not  appropriate  for  R&M  and  data  is  not  screened. 

I  think  the  Air  Force  has  failed  to  provide  the  required 
follow-up  of  the  projected  R&M  so  that  it  can  be  compared 
with  field  data- 

Definitions  and  criteria  differences  between  testing  and 
operational  field  contribute  to  the  variance  of  R&M  data. 

Manager  12 

The  difference  is  in  the  labs  and  real  world  environment. 
There's  a  lot  of  procedures  that  we've  agreed  to  during 
those  phases  [testing!  that  perhaps  misrepresent  the 
numbers.  It's  basically  Just  an  environment  difference  and 
the  fact  that  usually  in  our  lab  testing  we  tend  to  rule  out 
a  certain  amount  of  failures.  If  there  is  a  failure  and  a 
fix  is  identified  but  has  not  been  incorporated  and  an  item 
fails  a  second  time  for  the  same  cause,  we  don't  count  it. 
Out  in  the  real  world  if  an  item  fails  twice  for  the  same 
thing  you  count  it  twice. 

* 

There  is  a  difference  in  what  we're  measuring.  The  defini¬ 
tion  changes  from  DT&E  to  operational. 

Manager  13 

You  don't  have  the  normal  run  of  the  mill  Air  Force 
technician  on  the  system  during  testing.  You  use  hand 
picked  highly  qualified  technicians  with  lots  of  experience. 
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The  difference  in  R&M  measures  is  caused  by  the  difference 
in  the  maintenance  personnel's  skill  and  experience. 

Manager  14 

Some  of  the  difference  comes  from  problems  associated  with 
the  beginning  of  the  system  production  run.  equality) 

The  change  in  system  environment  from  the  sterile  lab  to  the 
field  contributes  in  that  the  system  receives  rougher 
handling. 


Manager  15 

I  think  you  can  take  cases  [that  show  a  problem!  but  take 
our  program  for  instance  of  some  6,000  different  systems, 
parts,  LRUs,  and  SRUs;  there  is  only  a  handful  not  meeting 
their  reliability  growth. 

One  of  the  big  problems  is  understanding  reliability  growth 
and  reliability  maturity.  That  is  that  your  going  to  get  to 
a  certain  maturity  at  a  certain  point  in  the  life  of  a 
system.  Almost  everything  we've  got  is  right  on  the 
maturity  curve  and  getting  better.  There  are  LRUs  that 
we're  going  back  and  fixing.  When  you  put  it  in  perspective 
it 's  not  that  bad. 

I  don't  think  Ca  lot  of  people!  understand  reliability 
growth.  They  expect  the  predicted  mature  values  to  occur 
before  system  maturity. 

We  lay  in  spares,  reasonably,  at  the  mature  level,  not  at 
half  or  a  quarter  of  the  mature  level,  so  we  don't  have  to 
buy  so  many.  So  in  the  mean  time  you  set  yourself  up  for  a 
shortfall  as  you  go  up  the  growth  curve. 

Half  the  problem  with  the  R&M  program  is  understanding  the 
def initions. 
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